{"id":"5c5a4ade-1b52-480d-b03a-5e7bb58fe9cc","arxiv_id":"2607.25431","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A per-commit multi-view repository-index system with incremental updates that match independent rebuilds at 8.7x/25.4x median speedups, and context policies that cut agent trajectory tokens 50-87% while holding localization within a declared margin.","lead":"CodeNib pre-builds a repository's search index, code embeddings, and symbol graph per commit, keeps each view updated as code changes, and serves them to coding agents through one runtime. Its headline numbers are large — 8.7x/25.4x faster incremental updates that exactly match fresh rebuilds, and 50-87% fewer agent trajectory tokens at preserved localization — but are explicitly restricted to the cases where its checks pass.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Q4's headline graph speedups are gated on exact equality with a fresh rebuild that the paper's own audit shows is nondeterministic for Rust/TS (fresh/fresh changed-scope exactness 3/4), making the 15/33 match count and 8.7x median dependent on a single noisy reference.","rationale":"The reader's weakest-assumption analysis identifies exactly the concern I find most load-bearing: the Q4 graph speedups are conditional on exact equality with an independent rebuild, and the paper's own audit shows that rebuild is not deterministic for Rust and TS/JS. Because the 15/33 match count and the 8.67x median are computed only on transitions that match a single fresh rebuild, oracle nondeterminism directly threatens the reproducibility of the headline maintenance speedup. The paper is unusually careful—it reports raw ratios, mismatch counts, edge F1, and serving agreement separately, and explicitly states that a cold live-LSP rebuild is not a deterministic oracle. Vector maintenance is less affected because exact ordered Flat replay is deterministic and 28/31 transitions pass; Q3 and Q5 are explicitly conditional on matched subsets and a stated margin. The concern does not warrant rejection: the system's qualitative claims and the Go/Python speedups may well stand if the concrete test shows fresh-fresh exactness is stable in those languages and match status does not flip across rebuilds. Since the reader already assigned CONDITIONAL and my analysis supports that same verdict, no adjustment is needed.","tokens_in":38266,"tokens_out":6564,"duration_ms":70697,"concrete_test":"Re-run the Q4 graph maintenance study using the protocol of Appendix F, but for each of the 33 transitions construct 5 independent fresh rebuilds of the target commit (same commit, profile, LSP/SCIP versions, protocol 21). Compute each incremental output's match status against all 5 references under Eqs. (1)-(2). If any transition's exact-match status flips across references, or if the set of 15 matching transitions changes, the match count is oracle-dependent. Also recompute the 8.67x median on transitions that match all 5 rebuilds and compare to the reported median. Separately run a same-commit fresh-vs-fresh exactness audit for the Go and Python repositories to confirm the oracle is deterministic for the languages that actually contribute passing transitions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central Q4 claim—'When outputs match an independent rebuild, graph and vector updates are 8.7x and 25.4x faster at the median'—depends on the independent rebuild being a reproducible ground truth. Appendix F's same-commit audit reports fresh-vs-fresh edge F1 of 99.75% and serving agreement of 99.35%, with changed-scope exactness on only 3/4 cases; all 18 Rust and TS/JS graph transitions fail the exact whole-graph/serving checks. This means 'match an independent rebuild' (Eqs. 1-2, Sec. 4.2) is not a stable predicate: a given incremental output can match one fresh rebuild and mismatch another, and a Rust/TS update with 99%+ fidelity is excluded from the speedup summary solely because the reference itself is noisy. The 15/33 match count and the conditional 8.67x median are therefore measured against a single draw of a noisy oracle, not a settled property of the system. The paper reports this limitation honestly and excludes mismatches from latency, but the headline speedup is still defined relative to that unstable reference; reproducing the experiment with a different fresh rebuild could yield a different 15/33 and a different median. This is the load-bearing weak spot in the paper's central quantitative contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CodeNib is a multi-view repository-context data system that materializes lexical, dense, and structural views per commit, maintains selected views incrementally, and serves ranked retrieval, symbol navigation, and bounded context to coding agents through a manifest-mediated runtime. The paper's main contributions are: (C1) a view compiler that links heterogeneous artifacts to repository-relative source ranges; (C2) view-specific incremental maintenance whose speedups are reported only when the updated artifact exactly matches an independently rebuilt target; and (C3) a cost-visible agent runtime with evaluated context-delivery policies. Across 100 snapshots and five evaluation questions, the headline results are: 8.7x/25.4x median graph/vector update speedups on matching transitions, a 4.72x median live/static latency ratio on the 63% of navigation requests whose normalized locations match, and 50–87% token reductions with localization preserved for selected policies across five agent models. The paper is notable for reporting null results (graph expansion), conditional claims (static/live compatibility), and explicit validity boundaries throughout.","tokens_in":38438,"tokens_out":7837,"duration_ms":84739,"significance":"If the results hold, CodeNib provides a principled data-systems blueprint for repository-context serving that separates ranked candidates, source locations, and prompt history instead of collapsing them into one abstraction. The paper is unusually careful in distinguishing quality, compatibility, update fidelity, latency, and token usage, and it reports conditional speedups only where output equality is verified offline. It also ships code/data with frozen dataset IDs and includes a same-commit rebuild audit, which strengthens reproducibility. The main quantitative claim—exact-output incremental maintenance with large speedups—is defensible in principle, but its central Q4 evidence depends on the stability of an independent-rebuild oracle that the paper itself shows to be nondeterministic for Rust and TS/JS. The Q5 'preserve localization' claim similarly rests on an operational margin that is a reporting threshold rather than an externally justified equivalence bound. These two points are the key risk to the paper's headline contributions.","major_comments":[{"comment":"The conditional graph speedup claim (8.67x median, 15/33 transitions) depends on the independent rebuild in Eqs. (1)–(2) being a deterministic oracle. Appendix F's same-commit audit reports fresh-vs-fresh edge F1 of 99.75% and serving agreement of 99.35%, with changed-scope exactness on only 3/4 cases, and all 18 Rust and TS/JS graph transitions fail the exact checks. Thus the 15/33 match count and the median are measured against a single draw of a nondeterministic reference; a different fresh rebuild could yield different matches and a different median. The paper honestly reports this limitation, but the abstract still presents the conditional speedup as the headline without noting that, for graph maintenance, the matched transitions are entirely Go/Python. Please (i) run multiple independent rebuilds and require equality with all (or report the distribution across rebuild draws), or (i","section":"§4.2, §9.5, Appendix F, Table 8"},{"comment":"The Q5 claim that selected context policies 'preserve localization' with 50–87% fewer tokens rests on the operational margin ε=0.05 applied to the lower bound of a paired 95% CI. This is a reporting threshold, not a pre-registered equivalence margin, as the paper acknowledges. However, the selection is margin-sensitive: for Haiku, the selected Eager arm has ΔAR@5 = −0.009 with lower bound −0.043, so a slightly stricter margin (e.g., ε=0.04) would change the selection. The headline token reduction is therefore contingent on a non-justified threshold. I recommend either justifying ε from an external operational requirement or presenting the results as a token–quality frontier and softening the 'preserve localization' wording in the abstract and conclusion.","section":"§4.3, §9.7"}],"minor_comments":[{"comment":"The panel appears twice in the submitted text (duplicate caption and identical plot). Please check the figure assembly.","section":"Figure 10(c)"},{"comment":"The phrase 'Reported protocols are 21/4' is ambiguous. Please spell out which protocol numbers are pooled and which are excluded, and why.","section":"Appendix F"},{"comment":"The sentence 'Qwen-4B/8B score means use n=99/98 at k′=30/50' could be clarified: state why the sample sizes differ for those operating points.","section":"§9.1"},{"comment":"The description of the lifecycle trace (20 sessions, 42 requests) is clear, but the initial paragraph says 'Materialize/load/mixed trace; isolated/resident' in Table 2; consider adding a cross-reference to §9.6 for the derived request count.","section":"§9.1"},{"comment":"The transition in Eq. (4) is well defined, but the claim that 'invocation j+1 consumes Ĥ_j' should be stated even more explicitly to avoid implying the rewrite is visible to the model before the next call; the surrounding text does clarify this, but a one-sentence restatement would help.","section":"§7.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is thorough and unusually honest in reporting null and conditional results, and the code/data release is a strength. The main barrier to acceptance is the Q4 oracle non-determinism: the headline graph speedup is measured against a single draw of a fresh rebuild that the paper itself shows is unstable for Rust/TS. This is fixable by adding a multi-rebuild analysis or by restricting the claim to language groups where the oracle is stable. The Q5 margin issue is secondary but should also be addressed. I recommend major revision rather than rejection, as the core architecture and evaluation framework are sound and the concerns are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is one of the most honestly scoped systems papers I've read in a while — every headline number ships with its own validity boundary in the same sentence. Second, the load-bearing caveat is real: the Q4 \"independent rebuild\" oracle is nondeterministic for Rust and TypeScript/JavaScript, which means the exact-match gate that produces the 8.67x graph speedup is partly a coin flip. The paper says so itself, but the consequence deserves more weight than it gets.\n\nWhat's actually new is the integrated design: a commit-anchored manifest over lexical/dense/structural views, view-specific maintenance with offline rebuild-equivalence gating, and per-operation validity boundaries in reporting. That evaluation contract is the real contribution — a template for how agent-context systems should be measured. The graph-expansion null is reported as a null, Q3 separates coverage from conditional latency, Q5 reports token savings only under a stated localization margin, and code plus frozen datasets with hashes are shipped. That is reproducible evidence, and I trust these numbers more than most.\n\nSoft spots, in proportion. The Q4 oracle problem: the same-commit audit shows fresh-vs-fresh changed-scope exactness on only 3/4 cases, and all Rust/TS graph transitions fail the exact gate. So for those languages \"match an independent rebuild\" is a single draw from a noisy reference; a different rebuild could move the 15/33 match count and the 8.67x median. The paper discloses this and excludes mismatches from latency, which is good, but the headline is still defined against that unstable predicate. I'd want robustness across multiple rebuild draws, or a tolerance-based comparison, before treating 8.67x as settled. The 25.44x vector number is on firmer ground — vector replay is deterministic — but three Rust rows fail exact ordered replay, so it's also narrower than the headline suggests.\n\nThe other issues are smaller. The Q5 margin (epsilon = 0.05) is self-defined and Qwen-9B passes by 0.001; Haiku's compact arm is backfilled; OPUS/Vertex/Haiku revisions are unversioned. The synthesis benchmark is generated and judged by the same OPUS alias. Q3 samples positions from the static graph, so the 63.2% match rate is conditional on static-favorable requests. No prior or production system runs on the same benchmarks. All disclosed, none fatal, but they cap how far the claims generalize.\n\nThis paper is for systems people building coding-agent context infrastructure and anyone designing evaluations for agent data systems. It deserves a serious referee — not a desk reject. I'd send it out with a clear request to harden Q4 (multiple rebuild draws or tolerance-based matching) and tighten Q5's selection rule, but the core is sound.","headline":"CodeNib is one of the more honest systems papers around, but the headline 8.7x graph speedup rests on an exact-match gate against a rebuild oracle that the paper itself shows is noisy for Rust/TS — that is the thing to fix before taking the number at face value.","tokens_in":39191,"tokens_out":3107,"would_cite":true,"duration_ms":33602,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coding-agent repository context can be served from reusable multi-view indexes that are incrementally maintained orders of magnitude faster than rebuilds, with latency and token savings tied to explicit output-match checks.","keywords":["repository context serving","coding agents","materialized views","incremental maintenance","symbol graph repair","vector reuse","static and live navigation","context policies"],"falsifier":"Rebuild the same eight repositories with a pinned compiler/language-server toolchain and run the incremental maintenance; if the Rust and TypeScript/JavaScript fresh-rebuild outputs are not bitwise reproducible across identical runs (as Appendix F's 3/4 changed-scope exactness suggests), then the 8.7x/25.4x conditional speedups cannot be attributed to the maintenance path alone. Conversely, if the match rate on a new, larger request sample for Q3 stays around 63% with the same per-language definition/reference split, the static provider's conditional latency advantage would be confirmed as a s","tokens_in":37942,"feed_emoji":"⚡","tokens_out":3916,"duration_ms":39469,"temperature":0.7,"pith_summary":"This paper argues that repository context for coding agents should be treated as a data-lifecycle problem: build reusable lexical, dense, and structural views once per commit, map every result to repository-relative source ranges, maintain each view through its own incremental path, and serve them through one runtime with a manifest recording what is available. The central evidence is that when incremental updates reproduce the output of an independent fresh rebuild, graph repair is 8.7x and vector reuse 25.4x faster at the median; static navigation matches live language-server results on 63% of requests and is 4.7x faster on that subset; and context policies deliver 50-87% token reductions while preserving localization across five models. The paper's distinctive move is to report these gains only under explicit validity boundaries — conditional on output matches — rather than as unconditional speedups.","feed_headline":"Incremental repo views beat rebuilds by 8.7x–25.4x","feed_subtitle":"CodeNib shows when outputs match an independent rebuild, updates are 8.7–25.4x faster and context tokens drop 50–87%.","key_machinery":"The central object is the repository manifest M_c and the multi-view materialization it controls: a lexical view (BM25/trigram), a dense view (FAISS embeddings), and a structural view (typed symbol graph with occurrence records), all aligned to repository-relative source ranges. The manifest records per-artifact status, commit, configuration, and capabilities, making stored state discoverable without a shared storage engine. The argument's load-bearing mechanism is the conditional output-match gate: incremental maintenance results (Eqs. 1-2) and static/live navigation equality (projection N) are verified offline against independent rebuilds or live servers, so reported speedups and latency s","core_discovery":"CodeNib's claim is that a multi-view repository serving system can make coding-agent context reuse measurable and economical. The key mechanism is the repository manifest M_c, which binds lexical (BM25/trigram), dense (embedding), and structural (symbol graph) views to a commit and to repository-relative source addresses, while leaving each view its own physical layout and update path. On the maintenance side, LSP-assisted graph repair and content-addressed vector reuse are compared offline against independent rebuilds, and speedups are reported only when the outputs match (Eqs. 1-2). On the serving side, static navigation is compared against live language servers on normalized path/start-li","pith_inferences":["If the conditional speedups hold in production, a practical recipe is to run incremental maintenance and fall back to rebuild when the offline match gate fails, combining 25x typical speedups with guaranteed exactness.","The 63% static/live match rate suggests a hybrid navigation service: route requests whose normalized location sets are predictable (e.g., definitions) to the static provider and keep live LSP for references; the workload-level latency saving would then depend on the request mix, not the conditional 4.7x.","The one-time compaction policy's model-dependent effect (123% tokens on Haiku vs 28% on Gemma) hints that context delivery should be per-model tuned rather than globally optimized — a claim the paper leaves open.","The same-commit rebuild audit (99.75% edge F1, but Rust/TS failures) suggests that the incremental-maintenance speedup numbers could shift if ground truth were measured against a more deterministic oracle; an editor would like to see the speedups re-reported on the subset of repositories where fresh rebuilds are exactly reproducible."],"forward_implications":["Incremental graph repair and vector reuse can make repository-context maintenance an order of magnitude cheaper than full rebuilds on transitions where outputs match (8.7x/25.4x median).","Static symbol navigation can absorb a majority of definition requests (80-99% match per language) and some references at 4.7x lower median latency, but references match only 22-73%, so live servers remain necessary.","Context policies with eager top-10 candidate injection plus one-time compaction preserve the localization margin at 50-87% fewer provider-reported trajectory tokens across five models.","The manifest's capability-driven loading lets one failed view (e.g., vector build) not invalidate successful siblings, so partial availability is explicit.","Operation-specific validity boundaries prevent conflating exact maintenance with projection-compatible navigation with policy-dependent delivery."],"fun_headline_variants":["Multi-view repo serving: updates match rebuilds at 8.7x–25.4x","CodeNib: repo context tokens drop 50–87% with preserved localization","Incremental repo views: 8.7x graph, 25.4x vector speedup on match","CodeNib: serving repo context, updates 25x faster when outputs align","Repo context policies cut tokens 87% while keeping localization"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole maintenance-speedup argument rests on treating an independent fresh rebuild as the deterministic ground truth for what incremental output should be; the paper's own audit shows fresh rebuilds are not exactly reproducible for Rust and TypeScript/JavaScript, so speedups are measured against a reference that is itself noisy for 2 of the 4 language groups.","fun_headline_variants_meta":{"raw":{"variants":["Multi-view repo serving: updates match rebuilds at 8.7x–25.4x","CodeNib: repo context tokens drop 50–87% with preserved localization","Incremental repo views: 8.7x graph, 25.4x vector speedup on match","CodeNib: serving repo context, updates 25x faster when outputs align","Repo context policies cut tokens 87% while keeping localization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000327,"raw_usage":{"total_tokens":1659,"prompt_tokens":729,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":818}},"tokens_in":473,"tokens_out":930,"duration_ms":9169,"temperature":1.0,"reasoning_tokens":818,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:28:53.207104+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the same eight repositories with a pinned compiler/language-server toolchain and run the incremental maintenance; if the Rust and TypeScript/JavaScript fresh-rebuild outputs are not bitwise reproducible across identical runs (as Appendix F's 3/4 changed-scope exactness suggests), then the 8.7x/25.4x conditional speedups cannot be attributed to the maintenance path alone. Conversely, if the match rate on a new, larger request sample for Q3 stays around 63% with the same per-language definition/reference split, the static provider's conditional latency advantage would be confirmed as a s","supporting_citations":[],"review_version":1}