{"id":"e1f777db-0f14-4eef-b7d4-89786a0f04d2","arxiv_id":"2607.23916","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An Inventory-as-Code governance loop propagates validation failures through a directed graph of AI agents and detects reasoning drift with a matched-bootstrap-calibrated Golden Path monitor.","lead":"This paper proposes a 'living inventory' system for managing risk in AI agents that act autonomously, replacing static model registries with a feedback loop. It formalizes how a failure in one agent should automatically block every downstream agent, and how to monitor for silent changes in an agent's reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Matched-bootstrap calibration in §6.4 resamples both the Golden Path and the production window, but monitoring uses a fixed baseline; the resulting δ is inflated and the advertised false-positive rate is not actually controlled.","rationale":"The paper's central empirical contribution is the trajectory monitoring protocol with its matched-bootstrap calibration. The reader's weakest assumption focuses on non-exchangeability of production streams (§9.1), which is explicitly conceded and is a limitation for deployment. My concern is more immediate and internal: even under the paper's own i.i.d. assumption, the bootstrap null in §6.4 is mis-specified. It treats both the Golden Path and the production window as random, but in operation the Golden Path is fixed at registration. Including Golden Path variability inflates the null variance, so the calibrated δ is too large and the actual false-positive rate is below the claimed α_fp. This undermines the 'necessity' claim for the matched bootstrap and the exactness of the FPR budget. It does not, however, invalidate the graph-propagation results (Proposition 5.1 is correct) or the general architecture; it points to a fixable statistical error. The reader's verdict of CONDITIONAL remains appropriate, since the empirical demonstration needs re-running with corrected calibration. I disagree with the reader about where the weakest assumption lies: it is not only the i.i.d. production assumption, but the construction of the null itself.","tokens_in":16144,"tokens_out":11676,"duration_ms":124422,"concrete_test":"Run a Monte Carlo null simulation: draw N_val=500 and N_prod=100 unit-sphere vectors from the same distribution; construct g from validation; for each of many seeds, calibrate δ via the paper's two-sample bootstrap and via a fixed-g bootstrap (resample p only at size N_prod); then compute the empirical FPR over independent production windows. If the two-sample method yields FPR significantly below the nominal 0.05 (e.g., ≤0.01) while the fixed-g method yields ≈0.05, the calibration is mis-specified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is the matched-bootstrap calibration (§6.4). The Golden Path g_vi is fixed at registration (Def. 6.1), and the drift metric D_t compares that fixed g_vi to a rolling production centroid p^(t)_vi (Def. 6.2). Under the null, the production window p is random but g is fixed; the correct null distribution of D_t for a deployed agent should be conditional on the observed g_vi. The paper's bootstrap instead resamples both g* (size N_val) and p* (size N_prod) from the validation corpus and sets δ to the 1−α_fp quantile of D(g*, p*). This null has variance ∝ 1/N_val + 1/N_prod, larger than the operational null's ∝ 1/N_prod, so δ is inflated. Consequently, the actual per-window false-positive rate is below α_fp, contrary to the claim that the matched bootstrap 'restores false-positive control at the intended α_fp' (§6.4, §8.4). The half-split criticism establishes only that the half-split null is too tight; it does not validate the two-sample resampling null. A correct matched bootstrap would keep g fixed and resample only p at size N_prod. The simulation (§8.4) cannot distinguish these because a single seed with zero soft flags is consistent with both a correct 5% FPR and a conservative FPR.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Inventory-as-Code governance loop for agentic AI systems, with four main contributions: a Degree of Autonomy (DoA) materiality score with a bounded tool-complexity term (Eq. 3), a DAG-based Composite Risk Propagation algorithm with a reachability-based blocking condition (Eq. 8), a trajectory drift monitor built on Golden Path centroids and a matched-bootstrap threshold (Eq. 11, §6.4), and practical protocols for LLM version changes, cycles, and prompt control. A synthetic 17-agent simulation (§8) is used to illustrate the framework: a single Tier 1 failure blocks exactly its transitive descendant set, and a matched-bootstrap calibration is claimed to cleanly separate in-distribution agents from drifted ones while the naive half-split null soft-flags roughly 90% of Tier 1 agents.","tokens_in":16572,"tokens_out":8411,"duration_ms":88876,"significance":"If the framework holds together, it provides a useful architectural template for extending model risk management to agentic systems: the graph-propagation logic is transparent, the DoA score separates tool breadth from impact/autonomy, and the trajectory-monitoring protocol tries to make drift thresholds a function of the production test statistic rather than an ad hoc cut. Strengths include explicit formulas, reproducible-simulation intent, and honest acknowledgement of limitations such as CoT unfaithfulness and non-i.i.d. production streams (§6.2, §9.1). However, the paper's central statistical claim about the matched bootstrap is not supported as written, the main 'theorem' is a restatement of the blocking definition, and the headline simulation is a single seed with no repeated-trial error analysis. The value is therefore more architectural than demonstrative at this stage.","major_comments":[{"comment":"The matched bootstrap resamples g* (size N_val) and p* (size N_prod) independently from the validation corpus. In production, however, D_t compares the registered, fixed Golden Path g_vi to a rolling production centroid p^(t). The correct null distribution is the distribution of D(g_vi, p*) conditional on the observed g_vi, whose variance scales as 1/N_prod. Resampling both centroids produces a null with variance scaling as 1/N_val + 1/N_prod. Since 1/N_val > 0, the threshold δ is inflated and the actual per-window false-positive rate is below α_fp. This contradicts the §6.4 claim that the matched bootstrap 'restores false-positive control at the intended α_fp'. The discrepancy is substantial in the paper's own parameter ranges (e.g., Tier 3 has N_val=50 and N_prod=100, doubling the variance scale). It also makes the Stage 2 threshold 1.5δ more conservative. The half-split comparison sho","section":"§6.4, Def. 6.2 and Eq. (11)"},{"comment":"The headline separation (10 Within Bounds, 0 Soft, 7 Hard) is one synthetic realization with a single seed and a single drift injection magnitude. The text itself notes that 'this seed realizes none' of the residual soft flags. With M=1000 bootstrap replicates but only one inventory configuration and one production realization, the simulation cannot provide evidence for 'false-positive control at the intended α_fp': a conservative calibration will also produce zero soft flags in one seed. Report repeated-seed distributions, or at least a Monte Carlo estimate of the achieved FPR under the null, and show the actual δ values under both the proposed two-sample bootstrap and a fixed-g conditional calibration.","section":"§8.4"},{"comment":"Proposition 5.1 is true but is a direct restatement of the blocking rule in Eq. (8): 'Blocked' is defined as having a contaminated ancestor, so the blocked set is exactly the union of transitive descendant sets by construction. The proof is a finite backward walk in a DAG. Since the paper itself says the guarantee 'holds by construction,' it should not be presented as a separate theorem or as an independent consequence of the composite risk formula. Reframe it as a design property of Eq. (8), and clarify that the β penalties carry no weight in the containment decision.","section":"§5.3, Proposition 5.1"}],"minor_comments":[{"comment":"The claim that 'tool breadth alone can never lift an agent above Tier 3' is conditional on α=0.6 and T_max=20. The paper notes T_max is a design parameter, but a short sensitivity statement for other choices would prevent overgeneralization.","section":"§4.4, Remark 4.1"},{"comment":"The term 'matched bootstrap' is misleading because g* and p* are drawn independently; consider 'size-matched two-sample bootstrap' or 'two-sample resampling null'.","section":"§6.4"},{"comment":"The drift injection magnitude of 0.60 as an 'orthogonal component' is not formally defined in the setup. Specify how the synthetic CoT embeddings are transformed under drift (e.g., addition of an orthogonal vector followed by renormalization).","section":"§8.4 / §8.1"},{"comment":"The certification criteria include a variance ceiling σ²_max and a 10% human-review requirement, but no procedure is given for setting σ²_max or auditing the review. Add operational guidance or cite a standard.","section":"§6.3"},{"comment":"The paper refers to a 'companion Python script' but provides no repository link, DOI, or checksum. Include an accessible reference for reproducibility.","section":"General"},{"comment":"The main text states the matched bootstrap 'restores false-positive control at the intended α_fp' without qualification, while §9.1 concedes that the i.i.d. bootstrap null 'can be optimistic' under non-i.i.d. production. These statements should be reconciled; the main-text claim should carry the same caveat.","section":"§6.4 vs §9.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the IaC concept is timely, but the calibration issue is substantive. The arXiv category math.NA also seems misaligned with the content, which is applied AI governance rather than numerical analysis. I would not reject on scope alone. The most important revision is to correct or reframe the bootstrap calibration in §6.4 and to replace the single-seed simulation with repeated-seed analyses, ideally including the achieved FPR under the fixed-g null. If those are addressed, the paper could be a useful practical contribution; in its current form the central statistical claim is not yet supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you work in model risk management for LLM systems. The paper proposes an Inventory-as-Code governance loop: a DoA materiality score with capped tool complexity, a DAG over agents with failure propagation, and a trajectory monitor that uses chain-of-thought embeddings to detect drift from a certified Golden Path. The framing is genuinely useful and the authors are unusually clear about what is and isn't established. The graph propagation result is correct but definitional—blocking is defined as reachability, so the blast radius theorem is true by construction. The DoA score is a sensible engineering answer to the tool-count dominance problem, and the two-stage soft/hard flag response is a practical improvement over a hard threshold. The paper also flags its own limitations, including autocorrelation in production streams and adversarial manipulation of CoT.\n\nThe soft spot is the calibration claim in §6.4. The paper says the matched bootstrap restores false-positive control by resampling both the Golden Path (N_val) and the production window (N_prod) from the validation corpus. But in production the Golden Path is fixed at registration; the only randomness in the drift metric under the null is the rolling window. By resampling g as well, the bootstrap null has variance proportional to 1/N_val + 1/N_prod instead of just 1/N_prod, so δ is inflated and the actual false-positive rate is below the advertised α_fp. The half-split criticism in the paper is valid—that null is indeed too tight—but it doesn't validate this two-sample resampling null. The simulation can't distinguish the two because a single seed with zero soft flags is consistent with both a correct 5% false-positive rate and a conservative one. This doesn't sink the framework, but the monitoring section needs a correct calibration: fix g and resample only p, or explicitly frame it as a two-sample test. The authors' own §9.1 acknowledges the i.i.d. bootstrap may be optimistic in real production, which is honest but doesn't cover this specific misspecification.\n\nCitations are appropriate and the companion script makes the simulation reproducible. I'd send this to peer review—a competent referee will catch the calibration issue, and the revision should strengthen the paper. I'd cite the DAG propagation and DoA ideas, not the drift calibration.\n\nRecommendation: engage with it for review, but expect the trajectory monitoring section to need rework.","headline":"A coherent and honest governance framework for agentic AI, but the matched-bootstrap calibration in §6.4 doesn't control the production false-positive rate as claimed.","tokens_in":17014,"tokens_out":3445,"would_cite":true,"duration_ms":33530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agentic AI can be governed by treating the inventory as a live graph in which a validation failure in one agent automatically blocks every agent that depends on it, directly or indirectly.","keywords":["model risk management","agentic AI","Inventory-as-Code","Directed Acyclic Graph","risk propagation","chain-of-thought embeddings","drift detection","matched bootstrap"],"falsifier":"On production or synthetic traces with temporal autocorrelation in in-distribution windows (e.g., slowly shifting market regimes), compute the empirical false-positive rate of the matched-bootstrap threshold at the stated α_fp; if it substantially exceeds α_fp, the exchangeability null is violated and the clean separation in §8.4 does not transfer. Also, an attempted adversarial input that preserves D_t < δ while altering reasoning semantics would falsify the monitor's claim to catch reasoning drift.","tokens_in":16046,"feed_emoji":"🕸️","tokens_out":3888,"duration_ms":37007,"temperature":0.7,"pith_summary":"The paper argues that traditional model risk management, built around static registries and point-in-time validation, is structurally obsolete for agentic AI systems whose behavior is path-dependent and self-directed. It proposes replacing the ledger with an Inventory-as-Code feedback loop: the inventory is a directed acyclic graph of agents, a failure in one node propagates along dependency edges to block exactly its transitive descendants, and a materiality score is designed so tool breadth alone cannot inflate an agent's tier. For drift, it treats an agent's Chain-of-Thought as a monitored artifact and calibrates a cosine-drift threshold with a matched bootstrap that mirrors the production sample sizes, avoiding the false-positive inflation of naive half-split calibration. The contribution is a single, internally consistent governance loop with a containment guarantee that holds by construction rather than by calibration.","feed_headline":"Agent failures can be contained to the exact downstream set","feed_subtitle":"New framework turns the model inventory into a live graph that auto-blocks only true dependents and flags drifting reasoning early.","key_machinery":"The load-bearing objects are (i) the agent dependency DAG G=(V,E) with ancestor/descendant sets; (ii) the Composite Risk Score R_C_j and its blocking condition Eq. (8), which sets Status=Blocked iff any ancestor is Failed or Blocked, making blast radius equal the transitive descendant set by construction; (iii) the Degree of Autonomy materiality score R = α(I×A) + (1−α)T with T capped at T_max=20, so tool breadth alone can never lift an agent above Tier 3; and (iv) the Trajectory Drift Metric D_t = 1 − cos(g,p) comparing a certified Golden Path centroid against a rolling production centroid, with threshold δ calibrated by a matched bootstrap that draws N_val and N_prod samples per replicate.","core_discovery":"The central claim is that agentic-system risk can be contained by three coupled mechanisms: a Degree of Autonomy materiality score with a capped tool-complexity term, a DAG-based Composite Risk Propagation rule whose blocking condition is pure reachability from any failed ancestor, and a Trajectory Monitoring protocol whose drift threshold is calibrated by a matched bootstrap that resamples a Golden Path and a production window at their actual sizes. The paper proves that the blast radius of a failure equals exactly the transitive descendant set of the failed agent, so quarantine is determined by graph contamination, not by an agent's inherent tier or by the penalty magnitudes. Simulation on","pith_inferences":["The reachability-based blocking rule is not specific to agentic AI: any governed system whose components compose in a DAG (data pipelines, microservice dependencies, supply chains) inherits the same containment guarantee.","The matched-bootstrap calibration principle extends beyond cosine drift to any monitoring statistic whose null variance depends on two sample sizes; naive half-split schemes will systematically mis-set thresholds whenever N_prod differs from N_val.","The paper's own §9.1 concedes that production streams may violate exchangeability; a natural extension is a block-bootstrap or time-aware null, and a sensitivity analysis of α_fp under autocorrelated drift.","Adversarial trajectory manipulation—where an attacker keeps D_t below δ while shifting reasoning semantics—is flagged as an open problem; the framework as proposed detects statistical displacement of stated reasoning, not semantic equivalence."],"forward_implications":["If a root agent fails, every downstream consumer, no matter how many hops away, is automatically blocked, and no unrelated agent is affected; the quarantine decision is independent of penalty magnitudes.","Tool count alone cannot game an agent into a higher tier, closing a dominance loophole in additive risk scores.","The matched bootstrap restores false-positive control: in-distribution agents stay below threshold, while genuinely drifted agents are hard-flagged, under the exchangeability assumption.","Two-stage response separates a human-review state from irreversible quarantine, so single benign exceedances do not take agents offline.","LLM base-model version changes trigger automatic Golden Path invalidation and revalidation, preventing silent baseline decay."],"fun_headline_variants":["Agent risk graph: failures only hit true dependents","Frame agent inventory as a DAG to auto-block only reachable failures","Drift detection calibrated by matched bootstrap avoids false alarms","Contain agent failures to exact downstream set via graph reachability","Agentic risk: quarantine by graph, not by tier"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The drift monitor's clean separation assumes production Chain-of-Thought embeddings are exchangeable with the certified validation corpus, so an i.i.d. matched bootstrap gives the true null distribution.","fun_headline_variants_meta":{"raw":{"variants":["Agent risk graph: failures only hit true dependents","Frame agent inventory as a DAG to auto-block only reachable failures","Drift detection calibrated by matched bootstrap avoids false alarms","Contain agent failures to exact downstream set via graph reachability","Agentic risk: quarantine by graph, not by tier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1383,"prompt_tokens":784,"completion_tokens":599,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":516}},"tokens_in":528,"tokens_out":599,"duration_ms":6229,"temperature":1.0,"reasoning_tokens":516,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:31:51.311127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On production or synthetic traces with temporal autocorrelation in in-distribution windows (e.g., slowly shifting market regimes), compute the empirical false-positive rate of the matched-bootstrap threshold at the stated α_fp; if it substantially exceeds α_fp, the exchangeability null is violated and the clean separation in §8.4 does not transfer. Also, an attempted adversarial input that preserves D_t < δ while altering reasoning semantics would falsify the monitor's claim to catch reasoning drift.","supporting_citations":[],"review_version":1}