{"id":"9a9366a8-804f-4ad5-acbc-222d383f7f7b","arxiv_id":"2608.00122","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A production deployment snapshot shows hook-based automatic capture and LLM curation turning 900 agent learnings into 1,144 shared question-answer memories, with no evidence yet of retrieval or coding-task benefit.","lead":"An enterprise system at SAP automatically captures lessons coding agents learn, asks a human to approve each one, and shares approved lessons with other agents through a vetted memory store. The report documents the architecture and first throughput numbers while explicitly leaving the question of whether the memory actually helps coding open.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Capture is not automatic: the paper's own §6 says the system only observes learnings externalized into configured memory files, so the recognition burden it claims to remove is merely deferred to the assistant.","rationale":"The reader's weakest assumption already identified the externalization dependency, and my analysis agrees. The paper is candid about it in Section 6, which is why this does not require a harsher verdict: the deployment story is internally consistent and the limitations section is unusually honest. However, the abstract and Section 1 frame capture as automatic and as removing the recognition burden; if the concern lands, the central novelty claim is overstated by one degree. A conditional verdict—accepting the system description while requiring evidence that the recognition burden is actually reduced—is the right level. I considered the security-gate circularity, but the paper explicitly discloses it and the central claim does not depend on a formal security guarantee. The deployment numbers are self-reported, but snapshot claims of this kind are not independently verifiable in general, and the authors explicitly defer relevance evaluations.","tokens_in":11563,"tokens_out":5562,"duration_ms":64850,"concrete_test":"Run a matched set of coding tasks with the Contributor Client either (a) configured with the AGENTS.md memory-writing instruction or (b) configured without it, holding all other settings fixed. Compare the number of contributed learnings in each arm. If the no-instruction arm captures zero (or near-zero) learnings, the system's capture depends entirely on the assistant being told to write memories, confirming that the recognition burden is deferred rather than removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim distinguishing this system from CQ is that contribution is 'automatically initiated' (§2.1) and that prior approaches require agents to 'recognize and explicitly record' lessons (§1). But the deployed hooks fire only when the assistant edits designated memory files; they do not extract lessons from ordinary coding activity. Section 6 explicitly concedes: 'the system only observes learnings that are externalized into the configured memory surface' and requires assistant-facing configuration (AGENTS.md / .claude) to make writing learnings part of the environment. Thus an agent that does not decide to write a memory produces no capture, regardless of how much task-adjacent experience occurred. The abstract's 'without agents deciding on their own to record lessons' is therefore not supported by the deployed design: the agent must decide to externalize a lesson; the platform automates only the subsequent diff/approve/curate steps. The 900-contribution snapshot measures a pipeline whose input depends on exactly the recognition behavior the paper claims to have removed. This is load-bearing because the claimed advance over CQ is automatic capture; if recognition is still a prerequisite, the system is a diff-triggered curation pipeline for hand-authored memory files rather than a platform-level replacement for agent judgment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes an ongoing production deployment of a shared organizational memory system for enterprise coding agents. The Contributor Client uses Claude Code hooks to detect changes to project-scoped Markdown memory files, diffs them, and asks the user for approval before sending candidates to a Curation Pipeline. The pipeline uses LLM agents to enrich approved diffs into question–answer memories, assign tool/type tags, run a security/privacy gate, check for collisions, and initialize trust scores. The Consumer Client serves the curated memories to coding assistants via MCP, with fast/normal/deep retrieval modes and a heuristic trust score. The deployment snapshot reports 900 contributed learnings, 483 productive learnings, 1,144 curated memories, and 7,863 semantic links across 35 tools/frameworks as of July 22, 2026. The authors explicitly state that retrieval relevance and downstream coding-task utility are still under evaluation.","tokens_in":11674,"tokens_out":4917,"duration_ms":60394,"significance":"If the architecture performs as claimed, the paper offers a concrete template for turning private, organization-specific development experience into a reusable, governed memory layer for coding agents. Its strengths are the explicit scoping of claims, the transparency of the limitations section (including the self-referential security-gate evaluation and the lack of trust-score calibration), and the level of implementation detail in the appendices. The reported numbers are internally consistent, and the paper does not overclaim downstream utility. The main scientific caveat is that the distinguishing claim of 'automatic capture' is materially weaker than stated in the abstract and Section 2.1, because the system only observes learnings that an assistant has already chosen to externalize into designated memory files.","major_comments":[{"comment":"The central claim distinguishing this system from CQ is that contribution is 'automatically initiated' and that prior approaches require agents to 'recognize and explicitly record' lessons. However, §6 concedes that 'the system only observes learnings that are externalized into the configured memory surface' and that the assistant must be instructed, via AGENTS.md or .claude, to write reusable learnings into memory files. This means the recognition burden is not removed; it is deferred to the assistant's decision to externalize a lesson. The 900-contribution snapshot therefore measures a pipeline whose input depends on exactly the behavior the paper claims to have made unnecessary. The abstract and §2.1 should be reworded to say the platform automates the diff/approval/curation steps after an assistant has written to a designated memory surface, not that capture happens without the assis","section":"Abstract, §2.1, §6"},{"comment":"The deployment snapshot reports 900 contributed learnings yielding 483 productive learnings and 1,144 memories, but it provides no denominator such as total agent tasks or total sessions. Without this, the numbers cannot assess the capture rate, which is the paper's core claimed improvement over CQ. The paper acknowledges 'current telemetry cannot identify the responsible pipeline stage' and that 'preliminary developer reviews' are not quoted or quantified; this should be stated in the main text rather than only in a passing sentence. At minimum, the paper should clarify that these figures are throughput, not capture effectiveness, and avoid any implication that a meaningful fraction of task-adjacent experience is being captured.","section":"§4 and Abstract"},{"comment":"The security-gate sanity check is reported in the main text as 80.4% appropriate, 19.0% cautious but non-blocking, 0.0% unsafe acceptances. The authors themselves identify the circularity: the system developers authored the synthetic cases, acceptable-action sets, and deterministic postconditions without independent adjudication. Because the main text presents these percentages before the limitation is disclosed, a reader could reasonably interpret them as evidence of gate effectiveness. The circularity should be stated immediately alongside the numbers in §2.2, and the main text should not present the result as more than a smoke test of the implementation.","section":"§2.2, Appendix D.1"}],"minor_comments":[{"comment":"There are spacing errors in the abstract (e.g., 'captureaplatform-levelpart' and 'task-adjacentexperience'). Please proofread the final PDF.","section":"Abstract"},{"comment":"The phrase 'Trust Score Initialization derives an initial reliability signal from examples, caveats, and recency' is unclear about how 'examples' are weighted; the full formula in Appendix F clarifies, but a forward reference would help.","section":"§2.2"},{"comment":"The snapshot mentions 'selected internal teams at SAP SE' but does not state the collection window (start date to July 22, 2026). Adding the time span would make the throughput figures interpretable.","section":"§4"},{"comment":"The trust-score calculation uses a prior mean C=0.5 and two different prior weights m=10 and m=3. These values are presented as fixed but no sensitivity analysis or justification is given. Since the paper explicitly says the score is uncalibrated, this is acceptable, but a sentence noting that these parameters are provisional would be honest.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The authors are unusually honest about the limitations, which is commendable. The core architecture is plausible and the deployment snapshot, while descriptive, is a legitimate form of evidence for a systems paper. The main concern is that the headline 'automatic capture' claim conflicts with the paper's own Section 6. If the authors revise the abstract and Section 2.1 to state the actual scope — automatic detection/approval/curation of externally written memory files — the contribution is still interesting, and the paper could be acceptable after minor further tightening. I recommend major revision because the mismatch is central to the paper's claimed advance over CQ."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is a carefully scoped engineering report on a deployed, three-stage organizational memory system for enterprise coding agents. The genuinely new piece is the capture mechanism: PostToolUse and Stop hooks mark local memory files dirty, diff against a replica, ask the user for approval, and send the patch to a curation pipeline. That is a real step past Mozilla CQ, which still expects agents to invoke query/propose/reflect tools. The deployment snapshot (900 contributions, 1,144 curated memories, 2.37 per productive learning) is self-reported but concrete and internally consistent.\n\nThe paper's honesty is a strength. It explicitly defers any claim about retrieval or coding-task utility, admits the security-gate evaluation is circular (the developers wrote the synthetic cases and postconditions), and flags that trust-score priors are uncalibrated. Those are real limitations, but they are disclosed rather than buried.\n\nThe soft spot is the headline claim of 'automatic capture.' Section 6 concedes that the system only observes learnings externalized into configured memory files; the assistant must be told, via AGENTS.md or .claude, to write reusable lessons there. So an agent that does not decide to record a lesson produces no capture. This means the recognition burden the abstract claims to remove is actually displaced to a configuration step. The platform automates the diffing, approval, curation, and retrieval, but not the decision that something is worth recording. The paper should have put that caveat in the abstract, not just in Limitations. It is not fatal — the pipeline is still a plausible improvement over CQ — but it undercuts the 'automatic' framing.\n\nThe other concerns are minor in context. The security-gate numbers are a sanity check, not a security benchmark. The trust-score and retrieval hyperparameters are hand-chosen but the paper says they are uncalibrated. The deployment numbers cannot be externally verified.\n\nWho gets value? Builders of agentic memory systems for internal codebases, especially ones that need to turn scattered notes into retrievable QA memories. It deserves a serious referee; a good review would push the authors to reconcile the abstract with Section 6 and maybe add a stage-level disposition log. I'd send it for review.","headline":"Deployment report with a real hook-based capture delta over CQ, but 'automatic capture' overstates a pipeline that still depends on agents writing lessons to memory files.","tokens_in":12351,"tokens_out":3827,"would_cite":false,"duration_ms":40347,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deployed system moves coding-knowledge capture from agent discretion to platform-level hooks, turning approved edits into reusable question-answer memories for future agents.","keywords":["enterprise coding agents","organizational memory","knowledge capture","question-answer memories","Model Context Protocol","curation pipeline","security gating","deployment snapshot"],"falsifier":"Run two matched teams through the same internal-DSL maintenance tasks, giving only one team retrieval from the shared memory store, and measure how often the same fix is rediscovered; if rediscovery rates are equal, the memory layer adds no measurable value. A cheaper check: add stage-level logging to see whether the 417 no-yield contributions are mostly dropped as duplicates or actually rejected for insufficient context, which would test the paper's tentative explanation.","tokens_in":11246,"feed_emoji":"🧠","tokens_out":4493,"duration_ms":50439,"temperature":0.7,"pith_summary":"The paper claims that enterprise coding knowledge can be captured as a platform-level side effect of normal coding work, rather than requiring agents to recognize and record useful lessons on their own. It describes a deployed pipeline in which a coding assistant's edits to designated memory files are detected by hook events, approved by the contributor, and turned by a curation layer into question-answer memories with security gating. These memories are then retrieved by future agents through a standard agent-data protocol, forming a shared organizational memory. The deployment snapshot counts 900 contributed learnings producing 1,144 curated memories, with 96.2% semantically linked, though retrieval quality and downstream utility are still being measured. If the capture model works, private development experience—internal DSL quirks, recent fixes, local conventions—can accumulate and be reused without repeated rediscovery.","feed_headline":"Coding agents bank team know-how without being asked","feed_subtitle":"Captured from approved edits, 1,144 memories now serve future coding agents.","key_machinery":"The load-bearing mechanism is the hook-driven Contributor Client, which uses the coding assistant's PostToolUse and Stop events to detect when a knowledge-base file has changed, compares the local knowledge base against a baseline replica, and generates a Diff Match Patch diff with five context lines and heading structure. This 'fly-on-the-wall' diff becomes a candidate learning that the human approves before it enters the curation pipeline. The second key object is the curated question-answer memory itself—a structured unit with a developer-facing question, direct answer, working example, caveats, tags, links, and trust score—which aligns stored knowledge with future problem-shaped queries.","core_discovery":"On its own terms, the paper's central claim is that making capture automatic changes where the recognition burden sits: instead of an agent deciding mid-task that a lesson is worth recording, the system watches for changes to project-local memory files, constructs a diff with surrounding context, and asks the human only for a yes/no approval. The curation pipeline then converts approved diffs into self-contained question-answer memories, runs deterministic security and privacy checks, assigns tool and type tags, checks for collisions, and initializes a trust score. A consumer client lets later coding agents retrieve these memories through the Model Context Protocol, either as quick tag-filte","pith_inferences":["The real recognition burden may shift from the agent to the configuration layer: because capture only sees learnings written into designated memory files, a developer or team must already instruct the assistant to externalize lessons there; enforcing this at scale is an open operational problem.","If retrieval evaluation shows utility, the same loop could generalize beyond code to other agentic artifact work, such as runbooks, incident postmortems, or data-pipeline definitions.","A testable extension would vary the explicitness of memory-writing instructions (e.g., with and without AGENTS.md guidance) and measure the resulting capture rate, quantifying how much of the 'automatic' capture depends on configuration."],"forward_implications":["Organizations can accumulate private development experience as a byproduct of coding work, without relying on agents to recognize the future value of a lesson.","The three-stage separation (collection, curation, consumption) allows each stage to be governed and tuned independently.","The 2.37 memories per productive learning suggests a single captured edit can yield multiple reusable units when split by topic.","The security gate's 0% unsafe acceptances across 168 synthetic executions suggests basic filtering can block obvious secrets and risks before persistence.","The trust-score prior damping means a few early votes cannot inflate a memory's credibility."],"fun_headline_variants":["Coding agents auto-bank team knowledge from edits","Approved diffs become Q&A memories automatically","Platform-level memory: agents learn from team edits","1,144 memories served to future agents automatically","Auto-capture turns team edits into agent memories"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system only captures lessons that the coding assistant writes into designated memory files, so if an assistant does not recognize a lesson as worth externalizing (or the team has not configured it to), nothing is captured.","fun_headline_variants_meta":{"raw":{"variants":["Coding agents auto-bank team knowledge from edits","Approved diffs become Q&A memories automatically","Platform-level memory: agents learn from team edits","1,144 memories served to future agents automatically","Auto-capture turns team edits into agent memories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3382,"prompt_tokens":635,"completion_tokens":2747,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":379,"completion_tokens_details":{"reasoning_tokens":2675}},"tokens_in":379,"tokens_out":2747,"duration_ms":18107,"temperature":1.0,"reasoning_tokens":2675,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:14:29.082126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run two matched teams through the same internal-DSL maintenance tasks, giving only one team retrieval from the shared memory store, and measure how often the same fix is rediscovered; if rediscovery rates are equal, the memory layer adds no measurable value. A cheaper check: add stage-level logging to see whether the 417 no-yield contributions are mostly dropped as duplicates or actually rejected for insufficient context, which would test the paper's tentative explanation.","supporting_citations":[],"review_version":1}