{"id":"25f99bf7-1383-4deb-bef5-81e76e27e125","arxiv_id":"2607.10508","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Confine the LLM to compiling plan edits over a deterministic, versioned dataflow executor that never calls the model, so research guarantees hold by construction.","lead":"This vision paper argues LLM research agents fail because every step is an unchecked stochastic call, and proposes fixing them by treating the project as a database: a deterministic dataflow the LLM may only edit, never run. If the design works, automated science could inherit database-style reliability, reuse, and auditability by construction.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"The by-construction claim is deferred: D1–D5 are stated, but the derivation that they deliver the eighteen requirements is withheld, so the central thesis is not yet demonstrated in this report.","rationale":"The reader correctly flags that the report withholds the guarantee walkthrough, prototype, and early results, and therefore assigns CONDITIONAL with MODERATE confidence. That is the right posture for a vision paper whose central “by construction” claim is not yet cashed out. My load-bearing concern is more precise: the missing derivation is the single point on which the strongest claim currently rests. The reader’s weakest assumption (DAG representability of open-ended research) is important for eventual practicality, but the paper’s own text already treats the guarantee walkthrough as the next required step (Section 4). Because that step is still absent, the verdict should stay CONDITIONAL; nothing in this pass strengthens or weakens it further. Agreement is partial because the reader emphasizes the representation premise while I treat the deferred derivation as the more immediate load-bearing gap for the claim as written. No formal verification or shipped artifacts exist in this version, consistent with the reader’s assessment. The design remains worth building and debating; acceptance of the by-construction thesis should wait on the promised full version.","tokens_in":17159,"tokens_out":628,"duration_ms":7256,"concrete_test":"When the full version appears, check whether it contains an explicit derivation (or machine-checkable sketch) that, for each of R1–R6, N1–N4, T1–T4, C1–C4, cites only D1–D5 plus the declarations (S1)–(S16) and shows the property holds on the augmentation-study example without additional LLM-behavior assumptions. If any tag still relies on an unstated premise or on the LLM “behaving well,” the by-construction claim does not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that confining the LLM to a plan-compiler role over a deterministic L1 executor, under five design rules D1–D5, makes reliability, non-waste, transparency, and collaboration hold by construction. The paper states the rules (Section 3.4) and maps requirements to mechanisms (Table 2), yet explicitly postpones the guarantee walkthrough that would show each of the eighteen tags follows from those rules on the running example (Abstract; end of Section 3.2; Section 4). Without that derivation, the claim remains an architectural assertion rather than a shown consequence. The reader’s weakest assumption (that open-ended research can be represented as a standing DAG with executable success predicates) is real, but secondary: even if the representation works, the by-construction transfer still needs the missing walkthrough. The deferred prototype and early results compound the gap. This is not a flaw in the diagnosis or the two-stratum idea; it is a missing link between the stated rules and the claimed guarantees.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"This vision paper diagnoses failures of LLM-based AI-Driven Research Systems (ADRS)—unreliability (non-idempotent answers, unbacked claims, silent staleness) and waste (re-execution, rewrite, context bloat)—as consequences of an architecture in which every step is an opaque stochastic LLM call. It proposes confining nondeterminism by placing the research project in a deterministic, versioned dataflow engine (L1: immutable content-addressed artifacts, operators, append-only registry/log, signature cache, field-level IVM) while restricting the LLM-plus-user to a stochastic plan compiler (L2) that may only emit DagEdits. Five boundary rules (D1–D5: determinism, append-only, confinement, idempotence, evidence gate) are claimed to make reliability, non-waste, transparency, and collaboration hold by construction. The manuscript supplies a four-property / eighteen-requirement taxonomy mapped to DBMS counterparts (Table 2), a typed design (S1–S16), a running augmentation-study example, and a comparison table (Table 3). Guarantee walkthrough, prototype, and research agenda are deferred to a full version.","tokens_in":17455,"tokens_out":1363,"duration_ms":24364,"significance":"If the confinement thesis holds, the paper offers a clean architectural answer to a timely problem at the agent–data-systems boundary: trust and economy for open-ended research automation without inspecting model internals. Strengths include a sharp root-cause diagnosis, a usable R/N/T/C requirement map (Table 2), an explicit type-level design that makes the stochastic boundary checkable (Figure 4), and a clear contrast with text-to-SQL, RAG, semantic operators, agent frameworks, and build systems (Figure 2, Table 1, §3.5). The evidence gate (D5) and dependency-closed view assembly (§3.3) are concrete mechanisms that prior agent loops lack. As a vision contribution these ideas are significant for the DB community even before a full prototype, provided the by-construction transfer is made credible.","major_comments":[{"comment":"Abstract and §4 state that the guarantee walkthrough deriving each of the eighteen tags from D1–D5 on the running example is deferred to a full version. The central thesis (confinement yields reliability/non-waste/transparency/collaboration by construction) therefore remains an architectural assertion rather than a shown consequence in this report. At minimum, a sketch for the load-bearing tags R2, R5, N1, N2, and C4—showing which declarations and rules force each—should appear here; without it the claim that “every guarantee follows from those rules” (§1) is not yet demonstrated.","section":"Abstract; §3.2 end; §3.4; §4"},{"comment":"The design assumes open-ended research can be represented as a standing deterministic DAG of operators over content-addressed artifacts, with NL goals compiled into versioned edits and executable success predicates (goal-as-artifact, §3.2; view from dependency closure, §3.3). Exploratory goals are acknowledged only as “provisional predicates” and open problem OP6. If many research steps cannot be quarantined this way without the LLM retaining run-time control, the by-construction guarantees do not transfer. The manuscript needs a clearer scope statement: which classes of research process are in scope, and what fails when the representation breaks.","section":"§1 thesis; §3.2 (goal artifact); §3.3; §4 OP6"},{"comment":"Table 3 marks “Ours” with design-time ✓ across nearly all rows while status-quo ADRS and related systems are scored from existing behavior. For a vision paper this is acceptable only if the table caption and body make the asymmetry explicit and avoid implying empirical parity. Several collaborative and transparent cells for prior systems are marked ×/◦ without citation-level justification; either ground those cells or narrow the table to the reliable/non-wasteful columns that the text actually argues.","section":"Table 3; §2.3"}],"minor_comments":[{"comment":"Figure 1’s “empty corner” is rhetorically effective but the axes (trust vs. cost/open-endedness) are not operationalized; a short caption note on what “trust by construction” means relative to sampling/self-check curves would help.","section":"Figure 1"},{"comment":"S4 lists runtimes {Exec, Eval, Diff, …} without defining Diff or the fixed-set discipline; a one-line semantics for each named runtime would make D1 easier to check.","section":"Figure 4, S4"},{"comment":"The EditCache key uses canon(request) and digest(view) (S14); the manuscript should state whether canon is required to be deterministic and collision-resistant for R2, or merely a best-effort normalizer.","section":"§3.2, S14"},{"comment":"References include several 2025–2026 arXiv/CIDR items; ensure final versions and stable links are used when available, and that the distinction from Berkeley’s “AI-Driven Research for Systems” (footnote 1) remains clear in the camera-ready.","section":"References; footnote 1"},{"comment":"Placeholder venue header “Conference’17, July 2017, Washington, DC, USA” should be removed or replaced.","section":"Running header"}],"recommendation":"major_revision","confidential_remarks":"This submission is explicitly a partial manuscript (“full version, in preparation”) that withholds the guarantee walkthrough and prototype. That honesty is welcome, but the venue should decide whether a design-only vision without even a sketch derivation of the eighteen requirements from D1–D5 meets the bar, or whether the authors should be asked to fold a condensed walkthrough into this report before acceptance. Novelty relative to build systems + agent frameworks is real at the boundary (D3/D5, EditCache, goal-as-predicate), but rests on that missing transfer argument."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a clean architectural vision from the DB side on why research agents fail and how to fix them by construction. The punchline is confinement: put the project in a deterministic, versioned dataflow engine (L1) and let the LLM-plus-user only edit that plan (L2); the executor never calls the model, and claims enter the record only via execution under five rules D1–D5.\n\nWhat is new is the specific seat for the LLM and the unified map. Table 1 and Figure 2 place prior couplings (text-to-SQL, RAG, semantic ops, agent frameworks) honestly; Table 3 shows that agent-first systems fill non-waste while workflows and build systems fill reliability, and almost nobody fills transparency or collaboration for a research process. The eighteen-requirement R/N/T/C taxonomy (Table 2), the typed declarations S1–S16, dependency-closed views with digests, EditCache, and the evidence gate are a coherent package. The running example (augmentation study) is used carefully to motivate each tag. Diagnosis of opacity, stale results, and waste is concrete and matches what people actually see.\n\nSoft spots are real but proportionate for a vision report that labels itself as such. The central claim—that D1–D5 deliver the eighteen guarantees by construction—is deferred to the full version’s walkthrough; this report states the rules and the map but does not derive each tag on the example. The weakest assumption is also real: that open-ended research can live as a standing DAG with versioned goals and executable success predicates without the LLM needing run-time control. No prototype or numbers appear here. Circularity is low; this is architecture, not fitted math. Citations are appropriate and the contrast with Make/Bazel/Nextflow is fair.\n\nThis is for people building agentic research systems, reproducibility tooling, or agent-first data systems. It deserves a serious referee and a reading-group slot as a design thesis, with the understanding that acceptance of the by-construction transfer waits on the deferred derivation and early results. I would engage: the framing is useful even if the full guarantees need the next paper.","headline":"Clear systems vision that confines the LLM to plan compiler over a deterministic project executor; the by-construction guarantees are asserted but not yet walked through in this report.","tokens_in":18097,"tokens_out":541,"would_cite":true,"duration_ms":5879,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Put the research project in a deterministic dataflow engine and make the LLM only a plan compiler, so trust holds by construction.","keywords":["AI-driven research systems","agentic data systems","determinism","dataflow","materialized views","incremental view maintenance","provenance","reproducibility"],"falsifier":"Build the two-stratum prototype on the paper’s augmentation study: if the same request over an unchanged project still yields different results, or if a reported accuracy can enter the record without an Eval-node execution, or if a small upstream code change cannot list exactly the stale downstream results, the by-construction claim fails.","tokens_in":18008,"feed_emoji":"🗄️","tokens_out":978,"duration_ms":9366,"temperature":0.7,"pith_summary":"Today’s LLM research agents can run a study from question to figures, but they cannot be trusted: the same ask yields different answers, reported numbers need not come from any run, upstream edits silently leave stale results, and work is re-done because nothing records what already finished. The paper argues these failures share one root—every step is an opaque stochastic LLM call—and that the fix is not to inspect the model but to confine it. Organize the project as a deterministic, versioned dataflow engine over immutable artifacts, like a query plan over materialized views. The LLM and user act only as a stochastic compiler that may edit that plan; the executor never calls the model, and nothing enters the stored record without an execution behind it. Five design rules at that boundary turn ordinary database machinery—versioning, provenance, incremental maintenance, memoization, cost-based scheduling—into guarantees that research is reliable, non-wasteful, transparent, and collaborative. The LLM should be the query compiler, never the executor.","feed_headline":"LLM should compile the research plan, never execute it","feed_subtitle":"Confine the model to plan edits; a deterministic dataflow engine makes trust hold by construction.","key_machinery":"The two-stratum model: L1 is a deterministic, versioned dataflow engine (operators over content-addressed artifacts, append-only registry and log, field-level staleness); L2 is the stochastic compiler whose only output is a plan edit; five rules (determinism, append-only, confinement, idempotence, evidence gate) force every guarantee to follow from architecture rather than LLM behavior.","core_discovery":"Reliability, non-waste, transparency, and collaboration for AI-driven research systems hold by construction when nondeterminism is confined: the project lives in a deterministic, versioned dataflow engine (L1), the LLM plus user is only a stochastic plan compiler (L2) that may edit that plan, the executor never calls the LLM, and results enter the record only via execution, under five boundary rules.","pith_inferences":["The same confinement pattern could apply beyond research agents to any long-horizon tool-using agent whose claims must stay bound to tool outputs.","Treating research intent as a versioned goal artifact with an executable success predicate opens a path to machine-checkable completion criteria for open-ended studies.","Assembling LLM context from the dependency closure of the current plan, rather than a linear transcript, is a general alternative to compaction and similarity-based memory for agent systems.","If the prototype validates the map, classical database techniques (MVCC, IVM, cost-based optimizers) become the default reliability layer for agentic scientific work."],"forward_implications":["Reported results become evidence-backed by construction: only outputs of deterministic evaluation nodes enter the project record.","Repeated requests over an unchanged project return identical results via edit and result caches keyed on canonicalized request plus view digest.","A change recomputes only invalidated work at field granularity; shared intermediates are memoized by content-hash signature.","The whole project is a small, versioned database instance that can be shared, queried, and peer-reviewed as read-only queries over the submitted record.","Researcher effort shifts from versioning, re-running, and verification to hypothesis formation and experimental choice."],"fun_headline_variants":["Confine LLM nondeterminism: compile research plans, never execute them","AI research as DBMS: LLM is stochastic compiler, executor stays deterministic","Confine agent nondeterminism so reliability holds by construction","LLM plus user edit only the plan; deterministic engine runs research","Research lives in versioned dataflow; LLM never enters the executor"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Open-ended research can be steered as a standing deterministic plan of operators over versioned artifacts, with natural-language goals compiled into plan edits and executable success checks, without the model needing run-time control.","fun_headline_variants_meta":{"raw":{"variants":["Confine LLM nondeterminism: compile research plans, never execute them","AI research as DBMS: LLM is stochastic compiler, executor stays deterministic","Confine agent nondeterminism so reliability holds by construction","LLM plus user edit only the plan; deterministic engine runs research","Research lives in versioned dataflow; LLM never enters the executor"]},"model":"grok-4.5","effort":"low","cost_usd":0.004088,"raw_usage":{"total_tokens":1318,"prompt_tokens":907,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":40880000,"prompt_tokens_details":{"text_tokens":907,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":341,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":907,"tokens_out":70,"duration_ms":3933,"temperature":1.0,"reasoning_tokens":341,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T11:10:46.572765+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Build the two-stratum prototype on the paper’s augmentation study: if the same request over an unchanged project still yields different results, or if a reported accuracy can enter the record without an Eval-node execution, or if a small upstream code change cannot list exactly the stale downstream results, the by-construction claim fails.","supporting_citations":[],"review_version":1}