Pith. sign in

REVIEW 3 major objections 5 minor 18 references

Put the research project in a deterministic dataflow engine and make the LLM only a plan compiler, so trust holds by construction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 11:10 UTC pith:ERJONTL6

load-bearing objection Clear systems vision that confines the LLM to plan compiler over a deterministic project executor; the by-construction guarantees are asserted but not yet walked through in this report. the 3 major comments →

arxiv 2607.10508 v1 pith:ERJONTL6 submitted 2026-07-11 cs.DB cs.AI

Confining Nondeterminism: AI-Driven Research Systems as DBMSs for Reliable, Non-Wasteful, Transparent, and Collaborative Research [Vision]

Kyoungmin Kim , Anastasia Ailamaki This is my paper
classification cs.DB cs.AI
keywords AI-driven research systemsagentic data systemsdeterminismdataflowmaterialized viewsincremental view maintenanceprovenancereproducibility
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Today’s LLM research agents can run a study from question to figures, but they cannot be trusted: the same ask yields different answers, reported numbers need not come from any run, upstream edits silently leave stale results, and work is re-done because nothing records what already finished. The paper argues these failures share one root—every step is an opaque stochastic LLM call—and that the fix is not to inspect the model but to confine it. Organize the project as a deterministic, versioned dataflow engine over immutable artifacts, like a query plan over materialized views. The LLM and user act only as a stochastic compiler that may edit that plan; the executor never calls the model, and nothing enters the stored record without an execution behind it. Five design rules at that boundary turn ordinary database machinery—versioning, provenance, incremental maintenance, memoization, cost-based scheduling—into guarantees that research is reliable, non-wasteful, transparent, and collaborative. The LLM should be the query compiler, never the executor.

Core claim

Reliability, non-waste, transparency, and collaboration for AI-driven research systems hold by construction when nondeterminism is confined: the project lives in a deterministic, versioned dataflow engine (L1), the LLM plus user is only a stochastic plan compiler (L2) that may edit that plan, the executor never calls the LLM, and results enter the record only via execution, under five boundary rules.

What carries the argument

The two-stratum model: L1 is a deterministic, versioned dataflow engine (operators over content-addressed artifacts, append-only registry and log, field-level staleness); L2 is the stochastic compiler whose only output is a plan edit; five rules (determinism, append-only, confinement, idempotence, evidence gate) force every guarantee to follow from architecture rather than LLM behavior.

Load-bearing premise

Open-ended research can be steered as a standing deterministic plan of operators over versioned artifacts, with natural-language goals compiled into plan edits and executable success checks, without the model needing run-time control.

What would settle it

Build the two-stratum prototype on the paper’s augmentation study: if the same request over an unchanged project still yields different results, or if a reported accuracy can enter the record without an Eval-node execution, or if a small upstream code change cannot list exactly the stale downstream results, the by-construction claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reported results become evidence-backed by construction: only outputs of deterministic evaluation nodes enter the project record.
  • Repeated requests over an unchanged project return identical results via edit and result caches keyed on canonicalized request plus view digest.
  • A change recomputes only invalidated work at field granularity; shared intermediates are memoized by content-hash signature.
  • The whole project is a small, versioned database instance that can be shared, queried, and peer-reviewed as read-only queries over the submitted record.
  • Researcher effort shifts from versioning, re-running, and verification to hypothesis formation and experimental choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same confinement pattern could apply beyond research agents to any long-horizon tool-using agent whose claims must stay bound to tool outputs.
  • Treating research intent as a versioned goal artifact with an executable success predicate opens a path to machine-checkable completion criteria for open-ended studies.
  • Assembling LLM context from the dependency closure of the current plan, rather than a linear transcript, is a general alternative to compaction and similarity-based memory for agent systems.
  • If the prototype validates the map, classical database techniques (MVCC, IVM, cost-based optimizers) become the default reliability layer for agentic scientific work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This vision paper diagnoses failures of LLM-based AI-Driven Research Systems (ADRS)—unreliability (non-idempotent answers, unbacked claims, silent staleness) and waste (re-execution, rewrite, context bloat)—as consequences of an architecture in which every step is an opaque stochastic LLM call. It proposes confining nondeterminism by placing the research project in a deterministic, versioned dataflow engine (L1: immutable content-addressed artifacts, operators, append-only registry/log, signature cache, field-level IVM) while restricting the LLM-plus-user to a stochastic plan compiler (L2) that may only emit DagEdits. Five boundary rules (D1–D5: determinism, append-only, confinement, idempotence, evidence gate) are claimed to make reliability, non-waste, transparency, and collaboration hold by construction. The manuscript supplies a four-property / eighteen-requirement taxonomy mapped to DBMS counterparts (Table 2), a typed design (S1–S16), a running augmentation-study example, and a comparison table (Table 3). Guarantee walkthrough, prototype, and research agenda are deferred to a full version.

Significance. If the confinement thesis holds, the paper offers a clean architectural answer to a timely problem at the agent–data-systems boundary: trust and economy for open-ended research automation without inspecting model internals. Strengths include a sharp root-cause diagnosis, a usable R/N/T/C requirement map (Table 2), an explicit type-level design that makes the stochastic boundary checkable (Figure 4), and a clear contrast with text-to-SQL, RAG, semantic operators, agent frameworks, and build systems (Figure 2, Table 1, §3.5). The evidence gate (D5) and dependency-closed view assembly (§3.3) are concrete mechanisms that prior agent loops lack. As a vision contribution these ideas are significant for the DB community even before a full prototype, provided the by-construction transfer is made credible.

major comments (3)
  1. [Abstract; §3.2 end; §3.4; §4] Abstract and §4 state that the guarantee walkthrough deriving each of the eighteen tags from D1–D5 on the running example is deferred to a full version. The central thesis (confinement yields reliability/non-waste/transparency/collaboration by construction) therefore remains an architectural assertion rather than a shown consequence in this report. At minimum, a sketch for the load-bearing tags R2, R5, N1, N2, and C4—showing which declarations and rules force each—should appear here; without it the claim that “every guarantee follows from those rules” (§1) is not yet demonstrated.
  2. [§1 thesis; §3.2 (goal artifact); §3.3; §4 OP6] The design assumes open-ended research can be represented as a standing deterministic DAG of operators over content-addressed artifacts, with NL goals compiled into versioned edits and executable success predicates (goal-as-artifact, §3.2; view from dependency closure, §3.3). Exploratory goals are acknowledged only as “provisional predicates” and open problem OP6. If many research steps cannot be quarantined this way without the LLM retaining run-time control, the by-construction guarantees do not transfer. The manuscript needs a clearer scope statement: which classes of research process are in scope, and what fails when the representation breaks.
  3. [Table 3; §2.3] Table 3 marks “Ours” with design-time ✓ across nearly all rows while status-quo ADRS and related systems are scored from existing behavior. For a vision paper this is acceptable only if the table caption and body make the asymmetry explicit and avoid implying empirical parity. Several collaborative and transparent cells for prior systems are marked ×/◦ without citation-level justification; either ground those cells or narrow the table to the reliable/non-wasteful columns that the text actually argues.
minor comments (5)
  1. [Figure 1] Figure 1’s “empty corner” is rhetorically effective but the axes (trust vs. cost/open-endedness) are not operationalized; a short caption note on what “trust by construction” means relative to sampling/self-check curves would help.
  2. [Figure 4, S4] S4 lists runtimes {Exec, Eval, Diff, …} without defining Diff or the fixed-set discipline; a one-line semantics for each named runtime would make D1 easier to check.
  3. [§3.2, S14] The EditCache key uses canon(request) and digest(view) (S14); the manuscript should state whether canon is required to be deterministic and collision-resistant for R2, or merely a best-effort normalizer.
  4. [References; footnote 1] References include several 2025–2026 arXiv/CIDR items; ensure final versions and stable links are used when available, and that the distinction from Berkeley’s “AI-Driven Research for Systems” (footnote 1) remains clear in the camera-ready.
  5. [Running header] Placeholder venue header “Conference’17, July 2017, Washington, DC, USA” should be removed or replaced.

Circularity Check

0 steps flagged

No circular derivation: architectural vision with deferred guarantee walkthrough, not a self-referential prediction chain.

full rationale

This is a vision paper proposing a two-stratum architecture (deterministic L1 dataflow executor + stochastic L2 plan compiler) and five design rules (D1–D5) from which reliability, non-waste, transparency, and collaboration are claimed to follow by construction. It does not fit parameters to data and then re-label them as predictions; it does not define a quantity in terms of itself; it does not import a uniqueness theorem from the same authors to force its design; and it does not rename a known empirical pattern as a new first-principles result. The mapping of eighteen requirements to database mechanisms (Table 2) and the contrast with prior LLM–DB couplings (Table 1, Figure 2, Table 3) are argumentative and comparative, not circular reductions. The paper explicitly states that the guarantee walkthrough, prototype, and research agenda appear in a full version in preparation (Abstract; Section 4), so the by-construction claim is asserted rather than fully derived here—but that is incompleteness of demonstration, not circularity of reasoning. Self-citations are absent as load-bearing premises; references are to external agent, build-system, and DBMS literature. Score 0 is therefore the correct outcome under the hard rules: no quoted step reduces a claimed prediction or first-principles result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 5 invented entities

The paper is a systems vision. It does not fit free parameters to data. Load-bearing content is domain assumptions about research automation plus invented architectural entities (two strata, five rules, evidence gate, EditCache, dependency-closed views). Claims that guarantees hold by construction rest on those design choices and on the deferred full-version walkthrough, not on external empirical constants.

axioms (5)
  • domain assumption Research pipelines can be expressed as a DAG of deterministic operators over immutable, content-addressed artifacts (code, data, models, figures, environments).
    Core modeling choice of L1 (Section 3.1, S1–S9); without it the DBMS analogy does not apply to open-ended research.
  • domain assumption An LLM-plus-user can usefully compile natural-language research requests into plan edits (add/rewire nodes, put artifacts, freeze subtrees, set goals) without needing run-time control of execution.
    Thesis and L2 design (Section 3.2, S12–S13); if plan-only control is too weak for real research, confinement fails.
  • domain assumption Deterministic runtimes (Exec, Eval, Diff, …) exist for the computational steps research agents currently perform via tools.
    S4 and D1; required for byte-identical rematerialization and evidence-backed results.
  • domain assumption Familiar database machinery (MVCC/versioning, provenance, IVM, memoization, cost-based scheduling, isolation) transfers to research dataflows once nondeterminism is confined.
    Stated throughout Sections 1–3 and Table 2; the paper’s guarantee story depends on this transfer.
  • ad hoc to paper Five boundary rules D1–D5 are sufficient for the eighteen tagged requirements to hold by construction.
    Section 3.4 asserts this; full derivation is deferred to the full version, so the sufficiency claim is paper-specific and not yet shown here.
invented entities (5)
  • Two-stratum ADRS (L1 deterministic record/executor + L2 stochastic plan compiler) no independent evidence
    purpose: Separate what is true/computed from what to try next; confine LLM nondeterminism.
    Central invented architecture of the paper (Figure 3, Section 3).
  • Evidence gate (D5) / Eval-only path for claims into the record no independent evidence
    purpose: Prevent unexecuted LLM assertions (e.g., '94% accuracy') from becoming project facts.
    Key mechanism for R5; defined in Section 3.2 and rule D5.
  • EditCache keyed by canon(request) and digest(view) no independent evidence
    purpose: Make repeated asks over unchanged projects replay the same plan edit without re-invoking the LLM (R2).
    S14; depends on deterministic view assembly.
  • Dependency-closed View assembly (full ArtifactRef vs Digest along graph structure) no independent evidence
    purpose: Replace growing chat transcripts and lossy compaction with version-pinned, re-expandable context.
    Section 3.3 and Figure 5; distinctive context-management design.
  • Eighteen-requirement R/N/T/C taxonomy mapped to DBMS counterparts no independent evidence
    purpose: State what an ADRS should guarantee and which DB techniques deliver each property.
    Table 2; organizes the vision and comparison to prior systems.

pith-pipeline@v1.1.0-grok45 · 21186 in / 3633 out tokens · 42343 ms · 2026-07-14T11:10:46.572765+00:00 · methodology

0 comments
read the original abstract

LLM agents that conduct research (proposing ideas, writing and running code, analyzing results) can already carry a study from research question to figures, yet cannot be fully trusted. The same question asked twice in a row returns different answers; the agent announces a number that no execution produced, and tool use does not prevent this, because nothing binds what the agent reports to what its tools returned; a small upstream change leaves downstream results silently stale, with no way to list which ones; and the agent re-runs preprocessing and rewrites code it has already produced. We argue these failures share one root: every step of today's agent loop is a stochastic LLM call whose internal state nobody, including the agent, can check. Rather than trying to see inside the LLM, we take a lesson from databases, which earn trust without being watched, because deterministic operators over well-defined state make their guarantees hold by construction. We propose organizing a research project the same way. The project lives in a deterministic, versioned dataflow engine (in effect, a query plan over materialized views), and the LLM, together with the user, is a stochastic compiler that may only edit that plan. The executor never calls the LLM; LLM output enters only as versioned code and data that the executor then runs, and any asserted result enters the record only with an execution behind it. Five design rules at this boundary turn familiar database machinery, from versioning and provenance to incremental maintenance and cost-based scheduling, into guarantees that make research reliable, non-wasteful, transparent, and collaborative. This report presents the diagnosis, the requirements, and the design; the guarantee walkthrough, a prototype, and the research agenda appear in the full version, in preparation. The LLM, we argue, should be the query compiler, never the executor.

Figures

Figures reproduced from arXiv: 2607.10508 by Anastasia Ailamaki, Kyoungmin Kim.

Figure 1
Figure 1. Figure 1: The corner an ADRS should occupy. A DBMS is [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Where the stochastic call sits in five LLM–database couplings. Orange = stochastic (the LLM), teal = deterministic, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The two strata on the running example, with each component labeled by its declaration (S1)–(S16) in Figure 4. All [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The design as sixteen declarations. (S1)–(S9) form [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: How each family manages the LLM’s context. (a) Chat agents re-send a transcript that only grows. (b) On overflow the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The three storage tiers of an LLM-based system, in [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 1 canonical work pages

  1. [1]

    Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. 2025. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. CoRRabs/2507.09089 (2025). arXiv:2507.09089 doi:10.48550/ARXIV.2507.09089

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.09089 2025

  2. [2]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei A. Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail?. InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Pro...

    2025

  3. [3]

    Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya A

    Audrey Cheng, Shu Liu, Melissa Z. Pan, Zhifei Li, Bowen Wang, Alex Krentsel, Tian Xia, Mert Cemri, Jongseok Park, Shuo Yang, Jeff Chen, Lakshya A. Agrawal, Aditya Desai, Jiarong Xing, Koushik Sen, Matei Zaharia, and Ion Stoica. 2025. Bar- barians at the Gate: How AI is Upending Systems Research.CoRRabs/2510.06189 (2025). arXiv:2510.06189 doi:10.48550/ARXI...

    doi:10.48550/arxiv.2510.06189 2025

  4. [4]

    E. F. Codd. 1970. A Relational Model of Data for Large Shared Data Banks. Commun. ACM13, 6 (1970), 377–387. doi:10.1145/362384.362685

    doi:10.1145/362384.362685 1970

  5. [5]

    Paolo Di Tommaso, Maria Chatzou, Evan W Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame. 2017. Nextflow enables reproducible computa- tional workflows.Nature biotechnology35, 4 (2017), 316–319

    2017

  6. [6]

    Timo Eckmann and Carsten Binnig. 2026. A Vision for Autonomous Data Agent Collaboration: From Query-by-Integration to Query-by- Collaboration. In16th Conference on Innovative Data Systems Research, CIDR 2026, Chaminade, CA, USA, January 18-21, 2026. www.cidrdb.org. https://vldb.org/cidrdb/2026/a-vision-for-autonomous-data-agent- Conference’17, July 2017, ...

    2026

  7. [7]

    Juraj Gottweis, Wei-Hung Weng, Alexander N. Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan,...

  8. [8]

    arXiv:2502.18864 doi:10.48550/ARXIV.2502.18864

    Towards an AI co-scientist.CoRRabs/2502.18864 (2025). arXiv:2502.18864 doi:10.48550/ARXIV.2502.18864

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.18864 2025

  9. [9]

    George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-SQL.VLDB J.32, 4 (2023), 905–936. doi:10.1007/ S00778-022-00776-8

    2023

  10. [10]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. InAdvances in Neural In- formation Processing Systems 33: Annual Conference on Neural Informati...

    2020

  11. [11]

    Parameswaran

    Shu Liu, Soujanya Ponnapalli, Shreya Shankar, Sepanta Zeighami, Alan Zhu, Shub- ham Agarwal, Ruiqi Chen, Samion Suwito, Shuo Yuan, Ion Stoica, Matei Zaharia, Alvin Cheung, Natacha Crooks, Joseph Gonzalez, and Aditya G. Parameswaran

  12. [12]

    In16th Conference on Innovative Data Systems Research, CIDR 2026, Chaminade, CA, USA, January 18-21, 2026

    Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First. In16th Conference on Innovative Data Systems Research, CIDR 2026, Chaminade, CA, USA, January 18-21, 2026. www.cidrdb.org. https://vldb.org/cidrdb/2026/ supporting-our-ai-overlords-redesigning-data-systems-to-be-agent-first.html

    2026

  13. [14]

    Andrey Mokhov, Neil Mitchell, and Simon Peyton Jones. 2018. Build systems à la carte.Proc. ACM Program. Lang.2, ICFP (2018), 79:1–79:29. doi:10.1145/3236774

    work page doi:10.1145/3236774 2018

  14. [15]

    Liana Patel, Siddharth Jha, Carlos Guestrin, and Matei Zaharia. 2024. LOTUS: Enabling Semantic Queries with LLMs Over Tables of Unstructured and Struc- tured Data.CoRRabs/2407.11418 (2024). arXiv:2407.11418 doi:10.48550/ARXIV. 2407.11418

    Pith/arXiv arXiv doi:10.48550/arxiv 2024

  15. [16]

    Matthew Russo and Tim Kraska. 2026. Deep Research is the New Analytics Sys- tem: Towards Building the Runtime for AI-Driven Analytics. In16th Conference on Innovative Data Systems Research, CIDR 2026, Chaminade, CA, USA, January 18-21,

    2026

  16. [17]

    https://vldb.org/cidrdb/2026/deep-research-is-the-new- analytics-system-towards-building-the-runtime-for-ai-driven-analytics.html

    www.cidrdb.org. https://vldb.org/cidrdb/2026/deep-research-is-the-new- analytics-system-towards-building-the-runtime-for-ai-driven-analytics.html

    2026

  17. [18]

    Michael Stonebraker, Xinjing Zhou, Peter Kraft, and Qian Li. 2026. Consistency and Correctness in Data-Oriented Workflow Systems. In16th Conference on Innovative Data Systems Research, CIDR 2026, Chaminade, CA, USA, January 18-21, 2026. www.cidrdb.org. https://vldb.org/cidrdb/2026/consistency-and- correctness-in-data-oriented-workflow-systems.html

    2026

  18. [19]

    Yu, and Jianfeng Gao

    Guiyao Tie, Jiawen Shi, Dingjie Song, Yixiao Huang, Ziji Sheng, Xueyang Zhou, Daizong Liu, Pan Zhou, Yongchao Chen, Ran Xu, Lifang He, Qingsong Wen, Manling Li, Cong Lu, Shuai Li, Pengtao Xie, Yixuan Yuan, Rui Meng, Lei Xing, Lichao Sun, Caiming Xiong, Philip S. Yu, and Jianfeng Gao. 2026. AutoResearch AI: Towards AI-Powered Research Automation for Scient...

    Pith/arXiv arXiv doi:10.48550/arxiv.2605.23204 2026