REVIEW 3 major objections 4 minor 27 references
A deployed system moves coding-knowledge capture from agent discretion to platform-level hooks, turning approved edits into reusable question-answer memories for future agents.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 01:14 UTC pith:ZZ56LJN2
load-bearing objection Deployment report with a real hook-based capture delta over CQ, but 'automatic capture' overstates a pipeline that still depends on agents writing lessons to memory files. the 3 major comments →
Shared Organizational Memory for Enterprise Coding Agents: System Design and Deployment Snapshot
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the paper's central claim is that making capture automatic changes where the recognition burden sits: instead of an agent deciding mid-task that a lesson is worth recording, the system watches for changes to project-local memory files, constructs a diff with surrounding context, and asks the human only for a yes/no approval. The curation pipeline then converts approved diffs into self-contained question-answer memories, runs deterministic security and privacy checks, assigns tool and type tags, checks for collisions, and initializes a trust score. A consumer client lets later coding agents retrieve these memories through the Model Context Protocol, either as quick tag-filte
What carries the argument
The load-bearing mechanism is the hook-driven Contributor Client, which uses the coding assistant's PostToolUse and Stop events to detect when a knowledge-base file has changed, compares the local knowledge base against a baseline replica, and generates a Diff Match Patch diff with five context lines and heading structure. This 'fly-on-the-wall' diff becomes a candidate learning that the human approves before it enters the curation pipeline. The second key object is the curated question-answer memory itself—a structured unit with a developer-facing question, direct answer, working example, caveats, tags, links, and trust score—which aligns stored knowledge with future problem-shaped queries.
Load-bearing premise
The system only captures lessons that the coding assistant writes into designated memory files, so if an assistant does not recognize a lesson as worth externalizing (or the team has not configured it to), nothing is captured.
What would settle it
Run two matched teams through the same internal-DSL maintenance tasks, giving only one team retrieval from the shared memory store, and measure how often the same fix is rediscovered; if rediscovery rates are equal, the memory layer adds no measurable value. A cheaper check: add stage-level logging to see whether the 417 no-yield contributions are mostly dropped as duplicates or actually rejected for insufficient context, which would test the paper's tentative explanation.
If this is right
- Organizations can accumulate private development experience as a byproduct of coding work, without relying on agents to recognize the future value of a lesson.
- The three-stage separation (collection, curation, consumption) allows each stage to be governed and tuned independently.
- The 2.37 memories per productive learning suggests a single captured edit can yield multiple reusable units when split by topic.
- The security gate's 0% unsafe acceptances across 168 synthetic executions suggests basic filtering can block obvious secrets and risks before persistence.
- The trust-score prior damping means a few early votes cannot inflate a memory's credibility.
Where Pith is reading between the lines
- The real recognition burden may shift from the agent to the configuration layer: because capture only sees learnings written into designated memory files, a developer or team must already instruct the assistant to externalize lessons there; enforcing this at scale is an open operational problem.
- If retrieval evaluation shows utility, the same loop could generalize beyond code to other agentic artifact work, such as runbooks, incident postmortems, or data-pipeline definitions.
- A testable extension would vary the explicitness of memory-writing instructions (e.g., with and without AGENTS.md guidance) and measure the resulting capture rate, quantifying how much of the 'automatic' capture depends on configuration.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes an ongoing production deployment of a shared organizational memory system for enterprise coding agents. The Contributor Client uses Claude Code hooks to detect changes to project-scoped Markdown memory files, diffs them, and asks the user for approval before sending candidates to a Curation Pipeline. The pipeline uses LLM agents to enrich approved diffs into question–answer memories, assign tool/type tags, run a security/privacy gate, check for collisions, and initialize trust scores. The Consumer Client serves the curated memories to coding assistants via MCP, with fast/normal/deep retrieval modes and a heuristic trust score. The deployment snapshot reports 900 contributed learnings, 483 productive learnings, 1,144 curated memories, and 7,863 semantic links across 35 tools/frameworks as of July 22, 2026. The authors explicitly state that retrieval relevance and downstream coding-task utility are still under evaluation.
Significance. If the architecture performs as claimed, the paper offers a concrete template for turning private, organization-specific development experience into a reusable, governed memory layer for coding agents. Its strengths are the explicit scoping of claims, the transparency of the limitations section (including the self-referential security-gate evaluation and the lack of trust-score calibration), and the level of implementation detail in the appendices. The reported numbers are internally consistent, and the paper does not overclaim downstream utility. The main scientific caveat is that the distinguishing claim of 'automatic capture' is materially weaker than stated in the abstract and Section 2.1, because the system only observes learnings that an assistant has already chosen to externalize into designated memory files.
major comments (3)
- [Abstract, §2.1, §6] The central claim distinguishing this system from CQ is that contribution is 'automatically initiated' and that prior approaches require agents to 'recognize and explicitly record' lessons. However, §6 concedes that 'the system only observes learnings that are externalized into the configured memory surface' and that the assistant must be instructed, via AGENTS.md or .claude, to write reusable learnings into memory files. This means the recognition burden is not removed; it is deferred to the assistant's decision to externalize a lesson. The 900-contribution snapshot therefore measures a pipeline whose input depends on exactly the behavior the paper claims to have made unnecessary. The abstract and §2.1 should be reworded to say the platform automates the diff/approval/curation steps after an assistant has written to a designated memory surface, not that capture happens without the assis
- [§4 and Abstract] The deployment snapshot reports 900 contributed learnings yielding 483 productive learnings and 1,144 memories, but it provides no denominator such as total agent tasks or total sessions. Without this, the numbers cannot assess the capture rate, which is the paper's core claimed improvement over CQ. The paper acknowledges 'current telemetry cannot identify the responsible pipeline stage' and that 'preliminary developer reviews' are not quoted or quantified; this should be stated in the main text rather than only in a passing sentence. At minimum, the paper should clarify that these figures are throughput, not capture effectiveness, and avoid any implication that a meaningful fraction of task-adjacent experience is being captured.
- [§2.2, Appendix D.1] The security-gate sanity check is reported in the main text as 80.4% appropriate, 19.0% cautious but non-blocking, 0.0% unsafe acceptances. The authors themselves identify the circularity: the system developers authored the synthetic cases, acceptable-action sets, and deterministic postconditions without independent adjudication. Because the main text presents these percentages before the limitation is disclosed, a reader could reasonably interpret them as evidence of gate effectiveness. The circularity should be stated immediately alongside the numbers in §2.2, and the main text should not present the result as more than a smoke test of the implementation.
minor comments (4)
- [Abstract] There are spacing errors in the abstract (e.g., 'captureaplatform-levelpart' and 'task-adjacentexperience'). Please proofread the final PDF.
- [§2.2] The phrase 'Trust Score Initialization derives an initial reliability signal from examples, caveats, and recency' is unclear about how 'examples' are weighted; the full formula in Appendix F clarifies, but a forward reference would help.
- [§4] The snapshot mentions 'selected internal teams at SAP SE' but does not state the collection window (start date to July 22, 2026). Adding the time span would make the throughput figures interpretable.
- [Appendix F] The trust-score calculation uses a prior mean C=0.5 and two different prior weights m=10 and m=3. These values are presented as fixed but no sensitivity analysis or justification is given. Since the paper explicitly says the score is uncalibrated, this is acceptable, but a sentence noting that these parameters are provisional would be honest.
Circularity Check
One admitted circular step: security-gate sanity check is self-referential by the authors' own admission; the capture-automaticity claim is a disclosed scope limitation rather than a technical circularity.
specific steps
-
other
[Section 6 (Limitations) and Appendix D.1]
"The security-gate sanity check has a related circularity limitation: the system developers authored its synthetic cases, acceptable-action sets, and deterministic postconditions without independent adjudication, so the reported agreement may reflect the gate's design assumptions."
The 0.0% unsafe acceptance and 80.4% appropriate outcomes are computed by running the gate against cases whose expected outcomes were authored by the same developers who implemented the gate. The ground truth is therefore not independent of the system under test; agreement measures self-consistency with the gate's design assumptions, not external correctness. The paper itself acknowledges this ('may reflect the gate's design assumptions'). This is a 'prediction' that reduces by construction to the gate's own rules.
full rationale
This paper is primarily a deployment snapshot, not a derived theoretical chain: the 900/483/1,144/7,863 figures are reported telemetry, not predictions fitted to that data. The central claim of automatic capture conflicts with Section 6's admission that 'the system only observes learnings that are externalized into the configured memory surface' and requires assistant-facing configuration such as AGENTS.md; however, that is a scope mismatch or unsupported generalization, not a definitional reduction or a fitted-input-as-prediction step, so I do not count it as circularity. The one clear circular step is the offline security-gate sanity check, which the authors themselves label as having a 'circularity limitation': the test cases, acceptable-action sets, and postconditions were authored without independent adjudication, so the gate's 0.0% unsafe acceptance rate is partly an artifact of the gate's own design assumptions. This is a genuinely load-bearing sub-claim because the paper uses it to support the gate's stated function, but it is confined to an auxiliary sanity check and is disclosed. The self-citation to Pulipaka et al. (which includes a co-author) is used only as background motivation for memory poisoning and is not load-bearing. Overall, central deployment claims remain independent content, so the score reflects one partial circularity rather than wholesale circular derivation.
Axiom & Free-Parameter Ledger
free parameters (8)
- Trust-score prior mean C =
0.5
- Trust-score prior weights m =
10 (agent), 3 (human)
- Trust-score component weights =
example 2, caveats 1, recency 1, agent feedback 4, human feedback 2
- Retrieval hybrid alpha =
0.5
- Q/A fusion weights =
question 0.7, answer 0.3
- Minimum similarity threshold =
0.3 (declared, not enforced)
- Recency decay schedule =
full point for 30 days, linear decay to 0 at 730 days
- Retrieval pool sizes and defaults =
top_x=5 (max 50); Normal pool max(20, 2*top_x); collision pool top 5
axioms (7)
- domain assumption Claude Code hooks (PostToolUse, Stop) fire reliably after file-writing actions and at task completion, and the Script can detect local knowledge-base changes by diffing against a replica.
- domain assumption The Enrichment Agent (Claude Sonnet 4.5, temperature=0) converts hunks into Q-A memories without introducing claims unsupported by the captured evidence.
- domain assumption In-repository rule-based checks are adequate proxies for OSV-Scanner, GuardDog, Gitleaks, Bandit, Semgrep, and Presidio for enterprise memory content.
- domain assumption The contributor understands what is in the prepared diff, and approval constitutes informed consent for sharing.
- domain assumption Retrieved memories are inspected, applied, tested, and reviewed by the consuming agent or its supervisor, so imperfect memories are caught downstream.
- domain assumption Diff Match Patch's textual representation is a stable, semantically adequate rendering of the edit for curation.
- domain assumption The embedding and reranking setup (gemini-embedding 1536-dim question/answer vectors, cohere-reranker) provides meaningful similarity for collision detection and retrieval.
invented entities (1)
-
Heuristic trust score (0-10)
no independent evidence
read the original abstract
Enterprise coding agents rely on tools and retrieval, yet enterprise knowledge often remains outside public training data and formal documentation: internal DSLs, proprietary platforms, local conventions, recent fixes, and tacit workflows. Existing knowledge interfaces expose stored resources but still depend on agents recognizing and explicitly recording lessons worth reusing, disconnecting capture from the coding workflow and leaving development experience repeatedly rediscovered. We report an ongoing production deployment of a shared organizational memory system that makes capture a platform-level part of coding work: it collects task-adjacent experience with contributor approval, curates it into reusable question-answer memories, gates obvious security and privacy risks, and retrieves memories for future agents. This short paper describes the deployed lifecycle and an operational snapshot. Effects on retrieval and coding tasks remain under evaluation.
Figures
Reference graph
Works this paper leans on
-
[3]
doi: 10.48550/arXiv.2605.14563. Bazel. Starlark language,
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.14563
-
[4]
Accessed 2026-07-13
URL https://bazel.build/rules/language. Accessed 2026-07-13. Cursor. Cursor documentation,
2026
-
[5]
Accessed 2026-07-13
URL https://cursor.com/docs. Accessed 2026-07-13. Data Privacy Stack. Presidio: Detection and de-identification of sensitive data,
2026
-
[6]
Accessed 2026-07-14
URL https: //github.com/data-privacy-stack/presidio. Accessed 2026-07-14. Datadog. GuardDog: Malicious package detection,
2026
-
[7]
URL https://github.com/DataDog/guarddog/ tree/v3. Accessed 2026-07-14. Shuzheng Gao, Wenhao Zeng, Zhaojian Yu, Jianqiao Wangni, Chaozheng Wang, Kai Cai, Shilin He, and Michael R. Lyu. SWE-MeM: Learning adaptive memory management for long-horizon coding agents.arXiv preprint arXiv:2606.28434,
Pith/arXiv arXiv 2026
-
[8]
SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agents
doi: 10.48550/arXiv.2606.28434. Gitleaks. Gitleaks: Detecting secrets in source code,
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.28434
-
[9]
Accessed 2026-07-14
URL https://github.com/gitleaks/gitleaks. Accessed 2026-07-14. Google. Diff match patch,
2026
-
[11]
Accessed 2026-07-13
URL https://www.terraform.io/language. Accessed 2026-07-13. Felienne Hermans, Martin Pinzger, and Arie van Deursen. Domain-specific languages in practice: A user study on the success factors. InModels in Software Engineering, pages 423–437,
2026
-
[13]
doi: 10.48550/arXiv.2512.13564. Dongming Jiang, Yi Li, Songtao Wei, Jinxin Yang, Ayushi Kishore, Alysa Zhao, Dingyi Kang, Xu Hu, Feng Chen, Qiannan Li, and Bingzhe Li. Anatomy of agentic memory: Taxonomy and empirical analysis of evaluation and system limitations.arXiv preprint arXiv:2602.19320,
-
[14]
doi: 10.48550/arXiv.2602.19320. 8 Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations,
-
[15]
URL https://arxiv.org/abs/2310.06770. Tue Le, Minh V. T. Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. SWE- EVO: Benchmarking coding agents in long-horizon software evolution scenarios.arXiv preprint arXiv:2512.18470,
-
[16]
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan
doi: 10.48550/arXiv.2512.18470. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval- augmented large language models.arXiv preprint arXiv:2305.14283,
-
[19]
io/specification/2025-06-18
URL https://modelcontextprotocol. io/specification/2025-06-18. Model Context Protocol. Reference servers,
2025
-
[20]
Accessed 2026-07-13
URL https://github.com/modelcontextprotocol/ servers. Accessed 2026-07-13. Mozilla AI. cq architecture, 2026a. URL https://github.com/mozilla-ai/cq/blob/main/docs/architecture. md. Accessed 2026-07-13. Mozilla AI. cq quickstart, 2026b. URL https://github.com/mozilla-ai/cq/blob/main/docs/quickstart.md. Accessed 2026-07-13. MozillaAI. cq: Anopenstandardfors...
2026
-
[21]
ChatGPT memory dreaming, 2026b
OpenAI. ChatGPT memory dreaming, 2026b. URL https://openai.com/index/ chatgpt-memory-dreaming/. Sidharth Pulipaka, Stanislau Hlebik, Leonidas Raghav, Sahar Abdelnabi, Vyas Raina, Ivaxi Sheth, and Mario Fritz. Hidden in memory: Sleeper memory poisoning in LLM agents.arXiv preprint arXiv:2605.15338,
-
[22]
doi: 10.48550/arXiv.2605.15338. PyCQA. Bandit: Securityissuedetectioninpythoncode,2026. URLhttps://github.com/PyCQA/bandit. Accessed 2026-07-14. Salesforce. Salesforce Object Query Language (SOQL),
-
[23]
Accessed 2026-07-13
URL https://developer.salesforce.com/ docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm. Accessed 2026-07-13. 9 SAP. ABAP keyword documentation, 2026a. URL https://help.sap.com/doc/abapdocu_816_index_ htm/8.16/en-US/ABENABAP.html. Accessed 2026-07-13. SAP. SAP Cloud Application Programming Model, 2026b. URL https://help.sap.com/docs/SAP_...
2026
-
[24]
Accessed 2026-07-14
URL https://github.com/semgrep/semgrep/tree/develop/cli/src/ semgrep/mcp. Accessed 2026-07-14. Shopify. Liquid reference,
2026
-
[25]
Accessed 2026-07-13
URL https://shopify.dev/docs/api/liquid. Accessed 2026-07-13. Upstash. Context7MCPServer,2026. URLhttps://github.com/upstash/context7. Accessed2026-07-13. Arie van Deursen, Paul Klint, and Joost Visser. Domain-specific languages: An annotated bibliography. ACM SIGPLAN Notices, 35(6):26–36,
2026
-
[26]
Yuntong Zhang, Yifan Li, Ge Li, Yutian Shi, and Yu Jin
URL https://papers.nips.cc/paper_files/ paper/2024/file/5a7c947568c1b1328ccc5230172e1e7c-Paper-Conference.pdf. Yuntong Zhang, Yifan Li, Ge Li, Yutian Shi, and Yu Jin. AutoCodeRover: Autonomous program improvement. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis,
2024
-
[27]
URL https://arxiv.org/abs/2404.05427. 10 This appendix retains the implementation and evaluation details needed to interpret the paper’s claims without reproducing full source schemas, prompts, fixtures, or the complete synthetic case manifest. Deployment-specific organization and infrastructure identifiers are replaced by neutral descriptions; model iden...
-
[2005]
doi: 10.1145/1118890.1118892. Microsoft. Kusto Query Language (KQL) overview, 2026a. URL https://learn.microsoft.com/en-us/ kusto/query/. Accessed 2026-07-13. Microsoft. Playwright MCP, 2026b. URL https://playwright.dev/docs/getting-started-mcp. Accessed 2026-07-13. Model Context Protocol. Specification, revision 2025-06-18,
arXiv 2026
-
[2009]
doi: 10.1007/978-3-642-04425-0_27. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo,ShihanDou,ZhihengXi,etal.MemoryintheageofAIagents.arXiv preprint arXiv:2512.13564,
-
[2023]
doi: 10.48550/arXiv. 2305.14283. Marjan Mernik, Jan Heering, and Anthony M. Sloane. When and how to develop domain-specific languages.ACM Computing Surveys, 37(4):316–344,
-
[2024]
URL https://github.com/google/diff-match-patch. Google. Antigravity documentation, 2026a. URL https://antigravity.google/docs/home. Accessed 2026-07-13. Google. OSV-Scanner: Vulnerability scanner for open source dependencies, 2026b. URL https: //github.com/google/osv-scanner. Accessed 2026-07-14. HashiCorp. The Terraform language,
2026
-
[2025]
com/engineering/equipping-agents-for-the-real-world-with-agent-skills
URL https://www.anthropic. com/engineering/equipping-agents-for-the-real-world-with-agent-skills. Anthropic. Dreams: Claude managed agents, 2026a. URL https://platform.claude.com/docs/en/ managed-agents/dreams. Accessed 2026-07-13. Anthropic. Effective harnesses for long-running agents, 2026b. URL https://www.anthropic.com/ engineering/effective-harnesses...
2026
-
[2026]
URL https://support.atlassian.com/jira-cloud-articles/ en-us/advanced-searching-using-jql/. Accessed 2026-07-13. SuyoungBae,JaehoonLee,ChangkyuChoi,YunSeokChoi,andJee-HyongLee. Rememberyourtrace: Memory-guided long-horizon agentic framework for consistent and hierarchical repository-level code documentation.arXiv preprint arXiv:2605.14563,
Pith/arXiv arXiv 2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.