Pith. sign in

REVIEW 4 major objections 6 minor 30 references

AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups

T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a locally hosted, open-weight RAG architecture organized around persistent project collections can capture and reuse a research group's tacit knowledge.

desk verdict A well-written architecture report that is honest about its own central gap: the tacit knowledge capture claim is explicitly deferred, so the paper is more of a systems description than a validated contribution. read the letter →

arxiv 2608.08883 v1 pith:TSGVJPCX submitted 2026-08-09 cs.AI

Jack Stark , Srinath Saikrishnan , Vikram Seenivasan , Bernie Boscoe , Andrew Lizarraga , Tuan Do This is my paper · ORCID
classification cs.AI
keywords retrieval-augmentedgenerationtacitknowledgeopen-weightmodelsmultimodalretrievalepisodicmemoryproceduralresearchcyberinfrastructurelocaldeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a research group's tacit knowledge—the unwritten how-to that lives in meeting notes, plotting standards, and analysis workflows—can be captured and reused by a locally hosted retrieval-augmented generation system built from open-weight models rather than commercial APIs. It presents architectural improvements to AquiLLM that organize interaction around a persistent project collection, add configurable semantic and episodic memory, load collection-scoped workflow instructions (called skills) into the chat prompt, and use salience-aware context packing to stay within limited GPU memory. If the approach works, research groups could retain expertise when members leave, keep sensitive data on their own infrastructure, and make informal practices inspectable and reproducible. The authors state plainly that the new retrieval components and the capture of tacit knowledge itself have not yet been rigorously evaluated.

What carries the argument

The load-bearing mechanism is skills, a form of procedural memory implemented as Markdown workflow instructions attached to a project collection and loaded into the session prompt only when that collection is active and its access permissions are verified. Around that sits a configurable memory layer that separates stable facts about the user from episodic semantic memories indexed for similarity search against prior message history. Salience-aware context packing carries the argument over the hardware constraint: deterministic token budgets, pinned sections, and staged pruning—tool-header deduplication, boilerplate collapse, extractive shortening, optional prompt compression, then salience-ordered dropping—keep prompts within GPU limits while guaranteeing inclusion of the latest turn, contiguous tool-call chains, and the last user turns.

What would settle it

Re-run the prior expert-rating study (141 responses over the 31-document astronomy corpus) with the new local retrieval stack and no external services; if faithfulness ratings, especially on comparative questions, fall below the earlier results, the claim that local control preserves answer quality is refuted. In a second check, ask new group members an onboarding question with and without the collection's skills loaded and look for the group's standards in the answers; no difference would show procedural memory is inert.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that the tacit-knowledge problem in small research groups can be treated as an architecture problem rather than just a model problem. AquiLLM makes the research group, not an individual notebook or shared knowledge base, the unit of shared context: project collections define document boundaries, carry their own workflow instructions, and persist user-specific facts and episodic conversation memories that can be retrieved by similarity search. The system layers procedural memory on top of the corpus so that how a group wants its documents read is bundled with the documents themselves. It replaces external retrieval services with local embedding and reranking, adds multimodal ingestion for plots and figures, and applies deterministic token budgets with staged pruning so long contexts fit in GPU memory. A prior expert evaluation of the system on astronomy material found that responses grounded in strongly localized evidence scored highest, while questions requiring synthesis across documents scored lowest, which motivates the memory and context-packing mechanisms introduced here.

Load-bearing premise

The load-bearing premise is that replacing external retrieval services with local embedding, reranking, and memory models does not silently degrade answer quality, and that the system actually captures tacit knowledge; neither has been measured.

Editorial extensions

If this is right

  • A research group can run the full pipeline—chat, embedding, reranking, transcription, and memory—on its own hardware, so papers, figures, and meeting minutes never leave local control.
  • Collection-scoped skills make onboarding documents, plotting standards, and reduction checklists the default frame for any chat over that project's corpus, while staying invisible to other projects.
  • Multimodal ingestion lets the system read plots and figures from slides or publications, so visual results enter the same retrieval and memory loop as text.
  • The memory layer carries user preferences and prior conversations across sessions, so follow-up work accumulates rather than restarting from scratch.
  • Deterministic token budgets and staged pruning keep long-context and multi-tool conversations within GPU memory, which is what makes local deployment on modest hardware feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the architecture works as intended, the unit a system answers from shifts from isolated documents to a living group corpus plus its unwritten rules; evaluation should then measure onboarding speed or answer fidelity for new group members rather than standard RAG benchmarks.
  • A direct test would compare answers with skills enabled versus disabled: if procedural memory is active, the system should measurably favor the group's own plot formats, reduction steps, and checklists over generic advice.
  • The paper's reported weakness in cross-document synthesis suggests that memory and context packing alone may not fix multi-source questions; a natural extension is to test whether explicit synthesis tooling closes that gap on informal artifacts such as Slack threads and meeting minutes.
  • Because local embedding and reranking are unmeasured, the pragmatic check is an A/B comparison against the previous external retrieval service on the same astronomy corpus; equal faithfulness would establish that local control costs nothing in answer quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents AquiLLM, an open-source, modular RAG-LLM framework intended for research groups, and describes a set of architectural and feature enhancements: local embedding and reranking with open-weight models, multimodal ingestion, vLLM-based OpenAI-compatible inference, a configurable semantic/episodic memory layer, collection-scoped skills, salience-aware context packing, and UI improvements. The authors argue that these enhancements support the capture and reuse of tacit knowledge in collaborative scientific settings while maintaining local control over data and infrastructure. No new quantitative evaluation is reported; the only evaluation cited is the authors' prior accepted paper [6], which assessed faithfulness on formal astronomy documents. Section 4 explicitly states that benchmarking on informal knowledge artifacts, such as meeting minutes, Slack messages, code comments, and playbooks, has not been done and that rigorously evaluating the capture and reuse of tacit knowledge remains to be done.

Significance. If the central claim were validated, AquiLLM would be a valuable contribution: a fully local, open-source RAG-LLM system with memory and skills could help small research groups retain informal expertise, protect sensitive data, and avoid vendor lock-in. The paper provides a detailed, internally consistent architecture description, a thoughtful comparison with related tools in Table 1, and honest discussion of failure modes from a prior evaluation. However, the significance is currently limited because the paper's headline claim—supporting tacit knowledge capture—is explicitly unevaluated, and the new retrieval and memory components are not benchmarked. The contribution is thus an architecture proposal with plausible design rationale rather than a demonstrated capability.

major comments (4)
  1. [Section 4] The central claim that AquiLLM supports tacit knowledge capture is explicitly deferred. The paper states: "we have not yet benchmarked our current approach on informal knowledge artifacts produced collaboratively by research groups such as meeting minutes, Slack messages, code comments, or collection-specific playbooks. Therefore, rigorously evaluating the capture and reuse of tacit knowledge remains to be done." Since the title, abstract, and Section 1 assert that AquiLLM supports tacit knowledge capture, the manuscript currently provides no evidence for its primary claim. The authors should either add a rigorous evaluation on informal artifacts or reframe the paper as a system-description/position paper in which tacit knowledge capture is a design goal, not an established property.
  2. [Section 3, local embedding and reranking] The new multimodal embedding and reranking models (Qwen3-VL-Embedding-2B and Qwen3-VL-Reranker-2B) are described as central improvements, but the manuscript states they "have not yet been fully evaluated in this deployment." No retrieval quality metrics, comparisons against the previous external services (e.g., Cohere), or ablation results are provided. Because retrieval quality is foundational to the claimed RAG improvements and to any downstream tacit knowledge capture, this missing evaluation is load-bearing.
  3. [Section 3, memory layer] The memory layer is a key mechanism for accumulating tacit knowledge, yet it is disabled by default, optional, and the authors note that it "introduces additional complexity around privacy and correctness, such as stale memories." No experiments, case studies, or user demonstrations show that semantic and episodic memory improves knowledge capture or retrieval. As described, the memory component remains a speculative design feature rather than a validated capability.
  4. [Section 2 and reference [6]] The only prior evaluation cited to support the system's effectiveness is [6], an accepted paper not included in this preprint and with overlapping authorship. That study assessed faithfulness on 141 responses over 31 formal astronomy documents, not on informal or tacit artifacts. Thus [6] cannot serve as evidence for the current paper's claim of supporting tacit knowledge capture, and its exclusion makes verification of even the formal-document results impossible for readers.
minor comments (6)
  1. [Abstract] The abstract contains "userinterfaceimprovements" with missing spaces; it should read "user interface improvements."
  2. [Section 3, chat model paragraph] The phrase "hybrid gated delta network" is used without a citation or explanation; either define the architecture or add a reference to a source that describes it.
  3. [Figure 1] Figure 1 is referenced in the text but is not visible in the provided manuscript; ensure the figure is included and legible, with a caption that names all labeled components.
  4. [Table 1] Table 1 gives a qualitative comparison with Open WebUI and NotebookLM, but no criteria or methodology are stated; clarify whether the comparison is based on documentation, hands-on testing, or authors' design intent.
  5. [Section 4] The Limitations section conflates known limitations with future work; consider separating unresolved limitations from planned extensions and adding a dedicated evaluation section if any benchmarking is included.
  6. [References] Several references are dated 2026 and may be future or in-press works; verify that all cited works (including the accepted paper [6]) are publicly available and correctly dated.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation found: the paper is an architecture report whose central tacit-knowledge claim is explicitly deferred, with only a minor overlapping-author self-citation.

full rationale

This paper is an architecture and deployment description rather than a derived prediction, so the standard circularity patterns do not apply: there are no equations, no fitted parameters, and no quantity is predicted from an input that already contains it. The only potentially load-bearing citation is [6], the authors' prior accepted study of AquiLLM faithfulness on 141 responses over 31 astronomy documents; it is used to motivate retrieval and context-packing design choices. While this is a self-citation by overlapping authors and the cited manuscript is not included in the preprint, it is an externally falsifiable evaluation rather than a definitional or numerical input, and the current paper does not derive its central tacit-knowledge claim from it. In fact, Section 4 explicitly states that 'rigorously evaluating the capture and reuse of tacit knowledge remains to be done,' and Section 3 concedes that the new local embedding and reranking models 'have not yet been fully evaluated in this deployment.' Those are evidence gaps, not circular reductions. The architecture description is self-contained as a design proposal; the central claim is under-supported rather than circular.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

This is a systems paper, so it introduces no fitted parameters and no invented entities. It rests on two domain assumptions: that prompt-injected skills and memory can effectively transfer tacit knowledge, and that locally hosted open-weight models are sufficient for scientific work. Both are stated but not tested here.

assumptions (2)
  • domain assumption Tacit knowledge in research groups can be captured and reused via collection-level Markdown instruction prompts (skills) and user-specific memory.
    Assumed in Section 3 under 'skills and skill packs' and the memory component; no evidence is provided in this paper that this representation actually captures tacit knowledge.
  • domain assumption Locally hosted open-weight models (e.g., Qwen3.6-27B) provide sufficiently faithful and capable inference for scientific research use without external commercial services.
    Motivated by prior work [6], but the current deployment with new components is not benchmarked, and Section 4 admits tacit-knowledge evaluation remains to be done.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups." pith.science (2026). https://pith.science/paper/TSGVJPCX

@misc{pith2026260808883,
  author       = {Pith},
  title        = {Pith review of: AquiLLM: An Architecture for Supporting Tacit Knowledge Capture in Research Groups},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TSGVJPCX}},
  note         = {Machine review of arXiv:2608.08883}
}
read the original abstract

Recent advances in retrieval-augmented generation (RAG) and large language models (LLMs) enable researchers to integrate AI into scientific workflows. However, using proprietary commercial AI systems raises concerns about transparency, reproducibility and privacy, which are essential for scientific practices. To this end, AquiLLM was developed as an open-source modular RAG-LLM framework using open-weight models, designed to support research groups in capturing tacit knowledge. In this work, we present a series of architectural improvements and feature enhancements to AquiLLM, including local embedding and reranking, multimodal capabilities, OpenAI-compatible inference interfaces, user interface improvements, semantic and episodic memory capabilities, and skills support. These enhancements were informed by discussions with domain experts, including astrophysicists and environmental researchers, and represent a step toward AI systems more closely aligned with scientific research practices.

Figures

Figures reproduced from arXiv: 2608.08883 by the authors.

Figure 1
Figure 1. Architecture of AquiLLM. Background workers handle document parsing and indexing into shared storage, while runtime requests pass through a prompt orches￾tration layer coordinating retrieval, memory, and tool use. Model inference is served locally via vLLM with separate components for chat, embedding, reranking, and mul￾timodal processing. and redundant retrieval work on constrained GPUs; per-request token and la￾te… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 18 canonical work pages

  1. [6]

    AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research

    Bernie Boscoe et al. “AquiLLM: Evaluating Faithfulness in Open-Weight RAG-LLM Systems for Scientific Research”. In:Proceedings of the 2026 US Research Software Engineer Conference (US-RSE). Accepted. San Jose, California, 2026

    work page 2026

  2. [1]

    Allen Institute for AI.More Than Open.url:https://allenai.org/ more-than-open(visited on 06/20/2026)

    work page 2026

  3. [2]

    AquiLLM.AquiLLM.https : / / aquillm . org/. Accessed: 2026-06-21. 2026

    work page 2026

  4. [3]

    GitHub repository

    AquiLLM.AquiLLM.https://github.com/AquiLLM/AquiLLM. GitHub repository. Accessed: 2026-06-21. 2026

    work page 2026

  5. [4]

    On the Dangers of Stochastic Parrots: Can Lan- guage Models Be Too Big?

    Emily M. Bender et al. “On the Dangers of Stochastic Parrots: Can Lan- guage Models Be Too Big?” In:Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency. FAccT ’21. New York, NY, USA: Association for Computing Machinery, Mar. 2021, pp. 610–623. isbn: 978-1-4503-8309-7.doi:10.1145/3442188.3445922.url:https: //dl.acm.org/do...

    arXiv 2021

  6. [5]

    arXiv:2108.07258 [cs]

    Rishi Bommasani et al.On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs]. July 2022.doi:10.48550/arXiv. 2108 . 07258.url:http://arxiv.org/abs/2108.07258

    arXiv doi:10.48550/arxiv.2108.07258 2022

  7. [7]

    AquiLLM: a RAG Tool for Capturing Tacit Knowledge in Research Groups

    Chandler Campbell, Bernie Boscoe, and Tuan Do. “AquiLLM: a RAG Tool for Capturing Tacit Knowledge in Research Groups”. In: arXiv:2508.05648 [cs]. US-RSE Conference, Oct. 2025.doi:10.48550/arXiv.2508.05648. url:http://arxiv.org/abs/2508.05648(visited on 10/25/2025)

    work page Pith review arXiv doi:10.48550/arxiv.2508.05648 2025

  8. [8]

    Carnegie Mellon University.Open Forum for AI (OFAI).url:https : //www.cmu.edu/engin/programs/ofai.html(visited on 06/20/2026)

    work page 2026

Show all 30 references
  1. [9]

    Harry Collins.Tacit and explicit knowledge. eng. Chicago, Ill: University of Chicago Press, 2013.isbn: 978-0-226-00421-1 978-0-226-11382-1

    2013

  2. [10]

    arXiv:2312.10997 [cs]

    Yunfan Gao et al.Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv:2312.10997 [cs]. Mar. 2024.doi:10 . 48550 / arXiv . 2312 . 10997.url:http : / / arxiv . org / abs / 2312 . 10997(vis- ited on 03/24/2026)

    2024 doi arXiv

  3. [11]

    Jetstream2: Accelerating cloud computing via Jetstream

    David Y. Hancock et al. “Jetstream2: Accelerating cloud computing via Jetstream”. en. In:Practice and Experience in Advanced Research Comput- ing. Boston MA USA: ACM, July 2021, pp. 1–8.isbn: 978-1-4503-8292-2. doi:10.1145/3437359.3465565.url:https://dl.acm.org/doi/10. 1145/34...

    2021

  4. [12]

    MorganHimesetal.Multi-Modal Masked Autoencoders for Learning Image- Spectrum Associations for Galaxy Evolution and Cosmology.arXiv:2510.22527 [astro-ph]. Oct. 2025.doi:10 . 48550 / arXiv . 2510 . 22527.url:http : //arxiv.org/abs/2510.22527

    2025 doi

  5. [13]

    arXiv:2501.06730 [cs]

    Edouardo Honig et al.Better Prompt Compression Without Multi-Layer Perceptrons. arXiv:2501.06730 [cs]. Jan. 2025.doi:10 . 48550 / arXiv . 2501.06730.url:http://arxiv.org/abs/2501.06730. 12 J. Stark et al

    2025 doi arXiv

  6. [14]

    Survey of Hallucination in Natural Language Generation

    Ziwei Ji et al. “Survey of Hallucination in Natural Language Generation”. In:ACM Comput. Surv.55.12 (Mar. 2023), 248:1–248:38.issn: 0360-0300. doi:10 . 1145 / 3571730.url:https : / / dl . acm . org / doi / 10 . 1145 / 3571730(visited on 03/26/2026)

    2023

  7. [15]

    Huiqiang Jiang et al.LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. en. Oct. 2023.url:https://arxiv. org/abs/2310.05736v2

    2023 arXiv

  8. [16]

    arXiv:2402.04647 [cs]

    Deqian Kong et al.Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference. arXiv:2402.04647 [cs]. Aug. 2025.doi: 10 . 48550 / arXiv . 2402 . 04647.url:http : / / arxiv . org / abs / 2402 . 04647

    2025 doi arXiv

  9. [17]

    arXiv:2309.06180 [cs]

    Woosuk Kwon et al.Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180 [cs]. Sept. 2023.doi: 10.48550/arXiv.2309.06180.url:http://arxiv.org/abs/2309.06180 (visited on 03/24/2026)

    2023 doi arXiv

  10. [18]

    From Stochas- tic Parrots to Digital Intelligence: The Evolution of Language Models and Their Cognitive Capabilities

    Andrew Lizarraga, Edouardo Honig, and Ying Nian Wu. “From Stochas- tic Parrots to Digital Intelligence: The Evolution of Language Models and Their Cognitive Capabilities”. In:WIREs Computational Statistics17.3 (2025),e70035.doi:10.1002/wics.70035.url:https://onlinelibrary. wil...

    2025 doi

  11. [19]

    arXiv:2402.17753 [cs]

    AdyashaMaharanaetal.Evaluating Very Long-Term Conversational Mem- ory of LLM Agents. arXiv:2402.17753 [cs]. Feb. 2024.doi:10 . 48550 / arXiv.2402.17753.url:http://arxiv.org/abs/2402.17753

    2024 doi arXiv

  12. [20]

    Latent Adaptive Planner for Dynamic Manipulation

    Donghun Noh et al. “Latent Adaptive Planner for Dynamic Manipulation”. en. In:Proceedings of The 9th Conference on Robot Learning. PMLR, Oct. 2025, pp. 2430–2448.url:https : / / proceedings . mlr . press / v305 / noh25a.html(visited on 03/28/2026)

    2025

  13. [21]

    Tejul Pandit et al.The Evolution of Reranking Models in Information Re- trieval: From Heuristic Methods to Large Language Models.arXiv:2512.16236 [cs]. Dec. 2025.doi:10.48550/arXiv.2512.16236.url:http://arxiv. org/abs/2512.16236

    2025 doi

  14. [22]

    Abenchmarkofexpert-levelacademicquestionstoassess AI capabilities

    LongPhanetal.“Abenchmarkofexpert-levelacademicquestionstoassess AI capabilities”. en. In:Nature649.8099 (Jan. 2026), pp. 1139–1146.issn: 1476-4687.doi:10 . 1038 / s41586 - 025 - 09962 - 4.url:https : / / www . nature.com/articles/s41586-025-09962-4(visited on 03/26/2026)

    2026

  15. [23]

    Polanyi.The Tacit Dimension

    M. Polanyi.The Tacit Dimension. Anchor books. Anchor Books, 1967. isbn: 978-0-14-341418-6.url:https://books.google.com/books?id= jwLXAAAAMAAJ

    1967

  16. [24]

    Model weights and configuration files for the post-trainedQwen3.6-27Bmodel.Apr.2026.url:https://huggingface

    Qwen Team.Qwen3.6-27B. Model weights and configuration files for the post-trainedQwen3.6-27Bmodel.Apr.2026.url:https://huggingface. co/Qwen/Qwen3.6-27B

    2026

  17. [25]

    Alec Radford et al.Robust Speech Recognition via Large-Scale Weak Su- pervision. en. Dec. 2022.url:https://arxiv.org/abs/2212.04356v1 (visited on 03/29/2026). AquiLLM Architecture for Tacit Knowledge Capture 13

    2022 arXiv

  18. [26]

    arXiv:1908.10084 [cs]

    Nils Reimers and Iryna Gurevych.Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. arXiv:1908.10084 [cs]. Aug. 2019.doi:10. 48550/arXiv.1908.10084.url:http://arxiv.org/abs/1908.10084

    1908 doi arXiv

  19. [27]

    Cognitive Architectures for Language Agents

    Theodore R. Sumers et al. “Cognitive Architectures for Language Agents”. In:Transactions on Machine Learning Research(2024). Also available as arXiv:2309.02427.url:https://openreview.net/forum?id=1i6ZCvflQJ

    2024 arXiv

  20. [28]

    Texas Advanced Computing Center.NSF Selects TACC Supercomput- ers for National AI Research Resource Pilot. Feb. 2024.url:https : //tacc.utexas.edu/news/latest- news/2024/02/01/nsf- selects- tacc-supercomputers-for-national-ai-research-resource-nairr- pilot/(visited on 06/20/2026)

    2024

  21. [29]

    Evans.Large language models in materials science and the need for open-source approaches.arXiv:2511.10673 [cs]

    Fengxu Yang, Weitong Chen, and Jack D. Evans.Large language models in materials science and the need for open-source approaches.arXiv:2511.10673 [cs]. Nov. 2025.doi:10.48550/arXiv.2511.10673.url:http://arxiv. org/abs/2511.10673

    2025 doi

  22. [30]

    JiayiYuanetal.The Science of Evaluating Foundation Models.arXiv:2502.09670 [cs]. Feb. 2025.doi:10.48550/arXiv.2502.09670.url:http://arxiv. org/abs/2502.09670. Acknowledgments.This work was supported by the Alfred P. Sloan Foundation (Grant No. G-2024-22720) and the National Sc...

    2025 doi arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.