{"id":"48f8998f-8f70-47b7-995c-bc80aa751229","arxiv_id":"2607.14390","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Git-bound, routed memory for coding agents, with retrieval and answer-assembly evaluations, achieves ~0.31 MRR on seed retrieval and up to 0.83 answer-sufficiency on rationale questions, at token costs three orders of magnitude below recorded history.","lead":"This paper argues that coding agents' memory should be built on git version control, and presents a system called Rekal that uses commit-linked session transcripts to answer developer questions. It introduces a three-mode routed architecture and evaluates it on real repositories, claiming token-efficient answers without manual labeling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Full router never evaluated end-to-end: reported 'routed' results omit decision synthesis, so the 382–980 token and 0.60–0.83 sufficiency claims don't yet apply to the integrated system.","rationale":"The paper is a genuine attempt to argue a strong architectural thesis with honest, if limited, empirical support. The reader's verdict of CONDITIONAL is appropriate: the evaluation has real gaps, but the direction is not invalid. My stress-test highlights a different but overlapping weakness: the paper's headline 'routed' numbers come from an ablated system that omits the synthesis mode, so the integrated three-mode router is never actually measured. This is not a fatal flaw — the components are each evaluated — but it directly affects the strongest quantitative claims (token cost and full-kind coverage). The reader's weakest assumption (session-to-commit linking accuracy) is also load-bearing, but the paper at least argues label noise is conservative; the router omission is a more immediate internal inconsistency. A concrete end-to-end router evaluation would settle the question. Since the reader already set CONDITIONAL and my concern reinforces that conditionality without shifting the verdict, UNCHANGED is the correct output.","tokens_in":10677,"tokens_out":4607,"duration_ms":49878,"concrete_test":"Run the full shipped skill (router + all three modes) on the same 30 test questions (15 per corpus), logging the router's mode choice for each question and measuring both answer-sufficiency and total tokens consumed under actual dispatch. Separately, sample the real query log to estimate the question-kind distribution p(kind) and compute the expected token cost. If the full-router sufficiency falls below 0.60 or expected tokens exceed 980 for the mixed distribution, the headline claim fails; if the full router matches or improves on the component floors within budget, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that the git-bound, routed system answers real developer questions at 382–980 tokens per question with sufficient context (0.60–0.83 sufficiency). However, the 'routed' condition in Table 2 is explicitly 'map + gated episodes' — it does not include decision synthesis, the third mode that the router (§7.4) dispatches for 'why' questions. Decision synthesis alone costs ~2.8–3.1k tokens per question (Table 2) and is the mode that achieves the headline 0.83 on Corpus B. Thus the 382–980 token figure is measured for a two-mode system, not for the full three-mode router described in the paper. Moreover, the router's triage decisions are never evaluated: there is no end-to-end run with the actual routing policy, no measurement of the real question-kind distribution, and the expected-cost formula is explicitly deferred as a 'future run.' This is an internal gap between the architecture and the evaluation: the abstract's token-economics claim ('Routed, the system answers at 382–980 tokens') is not supported for the system as designed. If the router sends a substantial fraction of real questions to synthesis, expected cost will be far above 980 tokens; routing errors could also lower sufficiency below the per-mode floors. The reader's linking-accuracy concern is valid, but this router-evaluation gap is more directly tied to the paper's headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the memory solution for the agentic development lifecycle is git itself: a post-commit hook links assistant sessions to the commits they produced, creating a self-labeling ledger (RekalBench), and a skill-layer router sends each question to one of three modes — a structural map, confidence-gated episodic recall, or decision synthesis. The seed-supply stage closes with a retrieval study on eight corpora (two with clean splits) under an incumbent-versus-candidate discipline, reporting that a tuned hybrid with a facet term reaches ≈0.31 pooled MRR versus grep floors, while five imported mechanisms are rejected. The answer-assembly stage reports per-kind answer-sufficiency results on two corpora (n=15 each), with decision synthesis reaching 0.83 on a young production corpus, and a two-mode 'routed' condition costing 382–980 tokens per question. The authors state that ground-truth labels are mined at zero annotation cost and that all artifacts are public.","tokens_in":11016,"tokens_out":6182,"duration_ms":58170,"significance":"If the results hold, the paper makes a useful conceptual contribution: treating git as the source of inherited guarantees (annotation, staleness, self-confirmation, containment) and distinguishing three question kinds with different answer-assembly mechanisms. The study is more methodologically disciplined than typical memory-system papers: pre-registered ship discipline, held-out splits, paired bootstrap CIs, explicit rejection of weak mechanisms, and publication of run records. The self-labeling benchmark, if its linking accuracy is validated, would be a substantial reproducibility contribution. However, the empirical support for the headline numbers is partly derived (best-config MRR), partially unmeasured (full-router token cost and sufficiency), and underpowered by the authors' own admission (n=15, one judge). The paper's significance is therefore conditional on closing these gaps; in its current form the abstract and conclusion overstate what is demonstrated.","major_comments":[{"comment":"The claimed token economics 'Routed, the system answers at 382–980 tokens per question' is not supported for the full three-mode system described in §7.4. The row labeled 'routed' in Table 2 is 'map + gated episodes'; it excludes decision synthesis, which is the mode the router dispatches for 'why' questions and which costs 2,762–3,135 tokens per question (Table 2, overall avg tok). The router's triage policy is never executed end-to-end, and the expected-cost formula is deferred to a 'future run' (§11). The abstract thus presents a two-mode cost figure as applying to the designed three-mode router. A concrete remedy is to run the full router on a held-out question set and report measured costs and sufficiency per routed question.","section":"§7.4, §8.3, Tables 2 and 4"},{"comment":"The gold labels for both studies are self-mined from commit–session links produced by the system's own post-commit hook. No independent validation of link accuracy or capture completeness is reported. The claim that label noise 'biases scores down, not up' (§6) is not proven; systematic misspecification (e.g., wrong-commit links, or unverbalized reasoning) could bias metrics in either direction. Since every MRR and sufficiency number inherits this gold, the linking precision is load-bearing. A concrete test is needed: human-validate a subsample of mined gold pairs and measure the hook's precision/recall against a manual session-to-commit mapping, then report the effect on the headline metrics.","section":"§6 with §5 and §8"},{"comment":"The answer-sufficiency evaluation has n=15 per corpus, one blind judge, and one execution model; the authors themselves state the pooled point estimates are 'not statistically separable' (§8.1) and that the honest summary is per-kind patterns, not separable pooled rankings. In this context the abstract's claim of 0.60–0.83 sufficiency and the conclusion's claim that the combination 'outperforms every single-mechanism alternative on coverage' (§1, §12) are not supported at the stated confidence level. The same limitation applies to the synthesis result on Corpus B (0.83 CI overlaps others). A second judge/agreement, larger n, or consistently hedged wording is required.","section":"§8.1–8.3, §11, §12"},{"comment":"The headline seed-stage number '≈0.31 pooled MRR' is not a directly measured held-out value. Table 1's footnote says 'best-config cells are derived from the tuned baseline plus the facet marginal,' and §5.2 says the marginal takes the hybrid 'to ≈0.31.' This assumes additivity of the facet marginal on the tuned baseline, which is not demonstrated; if the run record contains the actual end-to-end held-out MRR, it should be reported in the table. As published, the abstract's 'best configuration ≈0.31' and the derived '≈60×/≈15×' comparisons rest on an estimate, not a directly measured observation.","section":"§5.1–5.2, Table 1"}],"minor_comments":[{"comment":"The Conclusion's 'combination outperforms every single-mechanism alternative on coverage' is in tension with the Limitations section's 'per-kind patterns, not separable pooled rankings.' Recommend aligning the summary claims with the self-assessed strength of the evidence.","section":"§1, §12 vs §11"},{"comment":"The label 'routed' for 'map + gated episodes' is confusing given the three-mode router of §7.4; consider 'two-mode routed' or 'routed (no synthesis).'","section":"Tables 2 and 4"},{"comment":"References [9] and [10] use placeholder author strings ('Survey authors', 'State-contamination authors'); these should be replaced or corrected before publication.","section":"References [9], [10]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually transparent, and the reproducible artifacts are a real strength. However, the abstract and conclusion are more confident than the evidence: the full-router evaluation is missing, the self-labeling loop is unvalidated, and the sufficiency study is underpowered by the authors' own admission. These are fixable within the manuscript's scope, but they need to be addressed before publication. I would be willing to reconsider after an end-to-end router run (even on a modest held-out set), a subsample validation of the mined gold links, and damped comparative claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central claim — a git-bound, routed memory answering real developer questions at 382–980 tokens with sufficient context — is not actually measured for the full system. The \"routed\" condition in Table 2 is map + gated episodes only; decision synthesis, the third mode and the one that hits 0.83 on Corpus B, is kept separate and costs ~2.8–3.1k tokens. So the abstract's token figure is for a two-mode system, and the router's dispatch decisions are never evaluated end-to-end. That's a load-bearing gap, and the paper's own limitations section half-admits it by deferring the expected-cost figure to a future run.\n\nThat said, the paper earns a serious look. The git-bound thesis is a genuine design position: using the commit as ground truth, the merge as verification, and the ledger as an append-only store gives the memory system properties that bolted-on machinery has to fake. The self-labeling benchmark via commit-session links is clever and cheap, and the retrieval study is more disciplined than most of what crosses my desk — honest floors, held-out splits, paired bootstrap CIs, five imported mechanisms rejected. The facet term ported from SPM is a nice contribution, and the three-mode routing (breadth/pointed/rationale) is a useful way to decompose the problem. The paper also does the rare thing of stating its own weaknesses clearly — small n, single judge, self-labeled gold, directional point estimates.\n\nThe soft spots beyond the router gap: the headline MRR of ~0.31 is derived from the tuned baseline plus a facet marginal rather than measured directly; the \"label noise biases scores down\" claim is asserted, not proven, and false-positive labels could just as easily inflate scores; and the answer-sufficiency evaluation uses 15 self-authored questions per corpus with one automated judge. These are not fatal — they match the paper's own framing as a within-system characterization — but they mean the empirical support is thinner than the prose suggests.\n\nI'd bring this to a reading group: the architecture is worth arguing about, the evaluation is a good template for how to be transparent about limits, and the router gap is a useful teaching example of the difference between a designed system and a measured one. The flaws are addressable, and the direction is worth engaging with. Send it to peer review; a good referee could push the authors to run the full router end-to-end and validate the mining labels against human judgment.","headline":"The architecture is a real contribution, but the paper's headline token-economics claim is not measured for the full three-mode router — the routed condition omits decision synthesis.","tokens_in":11485,"tokens_out":1901,"would_cite":true,"duration_ms":21185,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the memory system for coding agents should be git itself — bound to version control, not bolted on — and that answering from it is a routed three-mode problem, not a single retrieval problem.","keywords":["agentic development lifecycle","git-bound memory","retrieval","answer sufficiency","decision synthesis","episodic memory","self-labeling benchmark","memory architecture"],"falsifier":"Manually audit a sample of the post-commit hook's session-to-commit links on a fresh corpus and compare the mined gold labels against human-verified labels; if a substantial share of links are wrong or missing, or if recomputing MRR and sufficiency on corrected labels moves the headline numbers materially, the central claim is falsified. Alternatively, run the same routed system on a corpus where the full reasoning was separately written down and check whether decision synthesis recovers what was recorded.","tokens_in":10539,"feed_emoji":"🧠","tokens_out":7381,"duration_ms":63994,"temperature":0.7,"pith_summary":"The paper argues that memory for coding agents should be bound to git rather than built as separate machinery: the commit supplies ground truth, rebuild supplies freshness, the merge supplies verification, and code review supplies containment. It then shows that answering from that ledger is not one retrieval problem but three modes — a structural map for breadth, confidence-gated episodes for pointed questions, and decision synthesis for why-questions — dispatched by a router. On a young production system of roughly 50k lines, the routed system reaches 0.83 answer-sufficiency on decision rationale and 0.60 overall, at 382–980 tokens per question, with ground truth mined from commit–session links at zero labeling cost. A sympathetic reader would care because this reframes agent memory as a version-control problem with guarantees inherited by construction, and because every result is replicable on any team's own git history.","feed_headline":"Three memory modes beat single-shot retrieval on real dev questions","feed_subtitle":"Git-bound ledger routes breadth, pointed, and why-questions at 382–980 tokens each.","key_machinery":"The load-bearing object is a git-bound ledger: an append-only store of parsed assistant session turns, tool calls, and commit SHAs, kept deliberately thin (no diff content; code is reconstructed from the SHA on demand), from which labels are mined by SQL over commit–session links. Standing on it are three memory modes — a structural map (condensed, regenerable subsystem description authored from the repository), confidence-gated episodic recall (seeds injected only when the retriever's own scores clear a calibrated bar), and decision synthesis (a decision-scoped gather of turns plus one synthesis call that reconstructs an evolved why-arc with per-claim commit pointers) — dispatched by an unt","core_discovery":"The central claim is that version control already supplies the four things memory machinery struggles to build: annotation (the commit links a session to the verified change it produced), freshness (the index is rebuilt from the append-only ledger and compiled structure is a function of the tree at a SHA), self-confirmation (only checkpoints landing on the default branch are exportable), and containment (code review is the only audited egress). On that ledger the paper separates seed supply from answer assembly: retrieval is closed as a study where five imported ranking mechanisms are rejected and two levers kept, while answer assembly requires a router that sends breadth to a structural map","pith_inferences":["The paper specifies but does not run a single expected-cost figure from the real question-kind distribution; instrumenting the router's own query log would turn the last cost table into E[tokens] = sum over kinds of p(kind) times cost(mode), a concrete next step.","Because the paper frames capture — how much reasoning agents verbalize — as the binding constraint, a natural test is to measure per-corpus verbalization rate against sufficiency; if raising verbalization raises sufficiency more than better retrieval does, the bottleneck claim is confirmed.","The confidence-gate signal (top-1 score and top-1–top-2 gap) is calibrated from the retriever's own scores, so the same trick could serve other agent-memory systems as a label-free injection gate.","The git-bound design could be stress-tested on older, larger histories to see whether the young-corpus advantage persists or whether stale episodes begin to poison synthesis over time."],"forward_implications":["A memory system for coding agents should be built into version control, inheriting annotation, freshness, verification, and containment from commits, rebuilds, merges, and review rather than constructing them separately.","Once transcripts are parsed into attributed turns, retrieval ranking is a solved-enough seed supplier: five imported re-ranking mechanisms failed, and the one gain came from an orthogonal facet evidence layer, not rank polish.","Answer assembly is not one retrieval problem: single-shot retrieval scores only 0.07–0.20 sufficiency on real questions, while routing by question kind reaches the per-kind floor at 382–980 tokens per question.","Ungated episode injection measurably degrades good answers, so confidence gating that can choose silence is a required component of the system.","Because ground truth is mined from commit–session links, the full benchmark and all scores are reproducible on any user's own history at zero labeling cost."],"fun_headline_variants":["Git-bound memory: router beats single-shot on dev Q&A","Zero-label git ledger answers dev questions at 382–980 tokens","Git as memory: 3 routes outrank single-shot retrieval on dev Q&A","Router on git ledger: 3 modes answer dev questions at 382–980 tokens","Zero-label git memory: 3 routes, 382–980 tokens per question"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole evaluation rests on the post-commit hook accurately and completely linking sessions to commits; if sessions are missed or linked to the wrong commit, the self-mined gold labels and every retrieval and sufficiency score are computed on a biased sample.","fun_headline_variants_meta":{"raw":{"variants":["Git-bound memory: router beats single-shot on dev Q&A","Zero-label git ledger answers dev questions at 382–980 tokens","Git as memory: 3 routes outrank single-shot retrieval on dev Q&A","Router on git ledger: 3 modes answer dev questions at 382–980 tokens","Zero-label git memory: 3 routes, 382–980 tokens per question"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001186,"raw_usage":{"total_tokens":4795,"prompt_tokens":865,"completion_tokens":3930,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3830}},"tokens_in":609,"tokens_out":3930,"duration_ms":27996,"temperature":1.0,"reasoning_tokens":3830,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T02:12:30.301527+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit a sample of the post-commit hook's session-to-commit links on a fresh corpus and compare the mined gold labels against human-verified labels; if a substantial share of links are wrong or missing, or if recomputing MRR and sufficiency on corrected labels moves the headline numbers materially, the central claim is falsified. Alternatively, run the same routed system on a corpus where the full reasoning was separately written down and check whether decision synthesis recovers what was recorded.","supporting_citations":[],"review_version":1}