{"id":"817ac5eb-76c0-4c60-999d-45f4ec01140a","arxiv_id":"2506.18257","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.","lead":"TableVault is a proposed file-based data management system for storing and tracking data tables created or modified by large language models. It is designed to add versioning, concurrency control, and provenance to LLM workflows so generated data can be rerun and audited safely.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reproducibility claim is internally contradicted: §3.1 admits 'natural output variability in language model responses,' so regenerating an LLM-generated table cannot ensure identical data artifacts. As stated, the central promise fails even if the implementation exists.","rationale":"I read the paper in good faith as a workshop system-description, not a benchmarked research contribution. Its central claim is a capability promise: concurrent execution, reproducibility, versioning, and composability. For that claim to hold, the described mechanisms must exist and, more fundamentally, the promised properties must be coherent. The most load-bearing weakness is not just the absence of code, which the reader correctly flagged, but an internal contradiction: the paper admits that LLM outputs vary naturally, so data artifacts generated by LLM calls cannot be exactly reproduced. That admission appears in §3.1 in the context of recovery after deletion: the authors say recovery is possible 'aside from natural output variability.' Since the central claim says TableVault ensures reproducibility, this caveat undercuts it from within, independent of implementation quality. The ACID claims in §3.2 are also asserted without demonstration—two-phase locking, write-ahead logs, and rollbacks for long-running external API calls would need substantial engineering and testing, and external side effects (e.g., API calls) are hard to roll back—but the reproducibility inconsistency is the sharper and more central problem, since it is acknowledged by the authors themselves. I therefore partially agree with the reader: the reader's weakest assumption was that the mechanisms exist and work, while my concern is that the reproducibility property is not well-defined for stochastic generation and is contradicted by the paper's own text. This supports the reader's rejection of the current version, so no verdict change is needed. The concrete test I propose would settle whether reproducibility holds in practice; absent such a test or a redefinition of the term, the central claim should not be accepted as stated.","tokens_in":7560,"tokens_out":3422,"duration_ms":44356,"concrete_test":"Obtain or run the TableVault implementation (the paper provides no repository, so if the authors cannot supply one, the claim remains untestable). Execute the same 'Generate Table Instance' operation twice from identical builder YAML files and identical dependency instances, with fixed LLM parameters such as temperature=0, deleting the first instance before regenerating. Compare the resulting dataframes with exact equality (e.g., pandas.testing.assert_frame_equal). Repeat for several tables. If any run produces different dataframes, 'ensuring reproducibility' fails as stated; also test the §3.1 recovery claim by deleting a table instance and rebuilding from saved builders, checking whether the recovered artifact matches the original.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that TableVault meets LLM-workflow demands by 'ensuring reproducibility,' and §1 further promises that 'data artifacts are reproducible for accuracy in automated workflows.' But §3.1 explicitly says that after deletion, 'all tables could be recovered if the external environment is preserved (aside from natural output variability in language model responses).' This is a direct, internal admission that re-running a generation workflow will not reliably reproduce the same data artifact. Because TableVault instances are populated by LLM API calls, even identical builders, identical dependency instances, and identical parameters can produce different dataframes across runs. The paper provides no mechanism—checkpointed outputs, deterministic decoding guarantees, or stored canonical results—that would make regeneration reproducible. Incremental materialization (§4.3) reuses unchanged columns from previous instances, but any LLM-generated column that is actually recomputed is subject to this variability. This is not merely a missing evaluation; it is an inconsistency between the central claim and the system description. A weaker claim—reproducibility of the workflow definition, provenance, and lineage—might be defensible, but the paper uses stronger language and does not define a notion of reproducibility compatible with stochastic model outputs. If reproducibility means exact artifact reproduction, the claim collapses; if it means process-level traceability, the paper should say so and revise the abstract and §1 accordingly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"TableVault is a proposed data management system for LLM-augmented workflows. The paper describes a file-based storage layout in which tables are folders, each containing timestamped instances; user-defined YAML 'builder' files specify how dataframe columns are generated, optionally via LLM API calls; TableString references express dependencies between table instances; standard operations (create, generate, delete, restart) run in background threads; write operations use logging, locking, rollback, and restart mechanisms; and instance generation supports incremental materialization by retaining columns whose dependencies and builders are unchanged. The abstract and introduction claim that TableVault supports concurrent execution, ensures reproducibility, provides robust data versioning, and enables composable workflow design.","tokens_in":7940,"tokens_out":3147,"duration_ms":36854,"significance":"If implemented and validated, the combination of provenance-tracking builders, explicit dependency references, and workflow-level concurrency would address a real and growing need in LLM pipeline management. The paper articulates useful concepts, especially the TableString syntax for expressing reduce, one-to-one, aggregation, convolution, and selection patterns, and the idea of incremental materialization to avoid redundant LLM calls. The contribution is nevertheless currently a design sketch: no implementation, code, formal proof, or experimental evaluation is provided, and the central reproducibility claim is internally contradicted by the paper's own acknowledgment of nondeterministic LLM outputs. The conceptual framework may be valuable as a position statement, but as a systems paper the guarantees asserted in Sections 2 and 3 are not established.","major_comments":[{"comment":"The central reproducibility claim is internally inconsistent. The abstract states that TableVault 'ensure[s] reproducibility,' and §1 promises that 'data artifacts are reproducible for accuracy in automated workflows.' However, §3.1 explicitly concedes that deleted tables can be recovered only 'aside from natural output variability in language model responses.' Because TableVault instances are populated by LLM API calls, regenerating a deleted instance from the same builders and parameters need not yield identical data artifacts. The paper offers no mechanism such as checkpointed outputs, deterministic decoding, or stored canonical results that would make regeneration exact. Either the reproducibility claim must be weakened to workflow definition and lineage reproducibility, or the system must store and reuse materialized outputs rather than regenerating them; as written, the claim collapses.","section":"§1 and §3.1"},{"comment":"The ACID and concurrency guarantees are asserted without supporting evidence. The paper states, 'We ensure ACID properties (Atomicity, Consistency, Isolation, Durability) using rollbacks, two-phase locking, and write-ahead logs,' but no formal specification, proof, implementation, system trace, or test is provided. The described execution flow also leaves open questions: external interrupts 'do not release acquired locks,' yet the paper does not explain how locked state is persisted across process restarts or how the 'restart' operation resumes from the last disk write without violating atomicity. For a system whose central value proposition is operational safety, these mechanisms are load-bearing and require at least a correctness argument, preferably backed by an implementation and tests.","section":"§3.2"},{"comment":"The incremental materialization optimization assumes that unchanged builders and unchanged dependencies imply unchanged columns. This assumption fails for LLM-generated columns because language model outputs are stochastic, as §3.1 acknowledges. Retaining columns from the previous instance only avoids recomputation when those columns are not regenerated; any column that must be computed again is subject to output variability, so the resulting dataframe may differ from what a full regeneration would produce. The paper does not define a consistency criterion under which partial materialization plus fresh LLM calls yields a valid instance, nor does it specify whether the system stores the original outputs to enforce determinism.","section":"§4.3"},{"comment":"The paper contains no experiments, benchmarks, or case studies. Claims that the design improves workflow efficiency, enables cost reduction, and supports complex RAG and document analysis patterns are illustrated only by examples and qualitative discussion. A comparison with existing systems such as Airflow, LangChain, DSPy, and DocETL is limited to a brief paragraph and does not identify measured advantages or trade-offs. As a result, the practical value of TableVault cannot be assessed from the manuscript.","section":"General"}],"minor_comments":[{"comment":"The text contains an unresolved placeholder, 'Figure ??[cite: 107, 137],' which should be replaced with an actual figure reference or removed.","section":"§4.2"},{"comment":"The phrase 'move the files to the destinated artifact folder' should read 'move the files to the destination artifact folder.'","section":"§4.1"},{"comment":"The DOI in the ACM reference format is a placeholder (https://doi.org/3735079.3735321) and must be updated to the final DOI.","section":"General"},{"comment":"The 'Restart TableVault Instance' operation is described in the table but not explained in the text; the relationship between restart and the lock-holding-on-interrupt behavior in §3.2 should be clarified.","section":"Table 1"},{"comment":"The example builder property table lists 'Changed Columns' but the text refers to 'generated column names'; using consistent terminology would improve readability.","section":"§4.1"}],"recommendation":"reject","confidential_remarks":"This is a short design/vision note with no implementation, code, or evaluation. Even for a workshop paper, the internal contradiction on reproducibility is fundamental; for the standards of a research journal, the absence of evidence for the ACID and concurrency guarantees is disqualifying. The authors could resubmit a revised manuscript that either clearly positions TableVault as a position/vision paper with no empirical claims, or provides an implementation and experiments that validate the guarantees and resolve the reproducibility issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a workshop-paper design proposal, not a validated system. The abstract promises ACID, reproducibility, and concurrency, but Section 3.2 asserts those properties without code, tests, or measurements, and Section 3.1 openly admits that regenerating LLM tables is subject to output variability. As the stress-test note says, the reproducibility claim collapses as stated. That is the load-bearing flaw, and it is internal to the text.\n\nWhat is actually good: the design has a coherent story. Versioned table folders, YAML builders, TableString references for lineage, and incremental column materialization form a useful vocabulary for managing LLM-produced data artifacts. The pattern table (reduce, one-to-one, aggregation, convolution, selection) is a nice compact abstraction, and the idea of reusing unchanged columns to avoid re-running expensive API calls is sensible. For a five-page workshop paper, the architecture is presented clearly enough that a reader can see what is being proposed.\n\nThe soft spots are real but concentrated. No implementation, no benchmarks, no formal proof of the ACID claims. The reproducibility contradiction is not a minor quibble: the central promise of the abstract fails unless the authors retreat to workflow-level provenance and say so explicitly. The unresolved figure placeholder also suggests the paper was rushed. However, none of this makes the design incoherent. It is a position piece that would be acceptable if the claims were reframed as design goals and implementation status were disclosed.\n\nWho gets value: people building LLM data infrastructure, especially those interested in versioning and lineage for agentic ETL pipelines. The paper gives them a concrete design to react to.\n\nRecommendation: send it to peer review — at workshop level, a well-scoped design proposal can be worth referee time — but require the authors to either ship the code and validate the claims or rewrite the abstract and Section 3 to explicitly describe what is implemented versus envisioned. I would not cite it as a result until either happens.","headline":"A clear design sketch for versioned LLM artifact management that overclaims in its abstract and has no implementation or experiments behind it.","tokens_in":8296,"tokens_out":1403,"would_cite":false,"duration_ms":18673,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TableVault applies database transaction methods to LLM-generated dataframes, claiming safe concurrent, reproducible, composable workflows.","keywords":["LLM agents","ETL systems","provenance","dataframes","document retrieval","data versioning","ACID transactions","workflow reproducibility"],"falsifier":"Kill TableVault mid-generation after some rows are written, restart it, and check whether the partially written instance is hidden and the last committed state is restored; if partial rows are visible, locks stay held, or the operation cannot resume under its original identifier, the ACID claim fails.","tokens_in":7372,"feed_emoji":"🗄️","tokens_out":7746,"duration_ms":81583,"temperature":0.7,"pith_summary":"TableVault is a proposed data-management platform for workflows in which large language models generate, transform, or analyze tables, documents, and other artifacts. The paper's central claim is that by borrowing classic database techniques—versioned storage, two-phase locking, write-ahead logs, rollbacks, and incremental materialization—such dynamic collections can support concurrent execution, reproducibility, and composable, transparent pipelines. A sympathetic reader would care because LLM-augmented data work is often run as ad-hoc scripts in notebooks, with lineage, crash recovery, and reuse left to the user. If TableVault works as described, it gives those workflows the same safety and provenance guarantees that databases have long provided for ordinary transactions.","feed_headline":"TableVault brings database transaction safety to LLM-built data tables","feed_subtitle":"Versioned, lock-protected table instances let LLM pipelines run concurrently and reproduce past outputs.","key_machinery":"The load-bearing mechanism is the versioned table instance combined with an explicit write-operation transaction protocol. Each table instance is defined by YAML builders (parameterized recipes for columns) and optional artifact files, stored as a timestamped folder; a metadata folder holds version dependencies, an operation log, and hierarchical shared/exclusive locks at table and instance granularity. The protocol wraps every write in an active log entry, saves the prior state, acquires locks, executes, and either commits (deleting the saved state and releasing locks) or rolls back, with locks deliberately retained across external interrupts so an operation can be resumed under the same identifier. Incremental materialization retains unchanged columns from earlier instances and executes builders only on empty entries, with builder dependencies ordered topologically.","core_discovery":"The paper introduces TableVault as a unified repository where each logical table is a folder of versioned instances, and each instance is produced by a user-written YAML “builder” that parameterizes Python functions or LLM API calls. A TableString reference lets one instance draw on another's dataframe as a dynamic input, with index forms encoding common transformations such as reduce, one-to-one, aggregation, convolution, and selection. The core claim is that every write operation—create, delete, generate—runs through a transactional flow that saves state, acquires hierarchical shared/exclusive locks, logs the operation, and on failure rolls back, thereby providing ACID (atomicity, consistency, isolation, durability) guarantees even though executions can be long and user-interruptible. Together the builders, TableStrings, and instance folders make data lineage explicit and allow regeneration of any instance from its recorded parameters.","pith_inferences":["A natural next test is to quantify saved LLM cost versus staleness risk: TableVault's incremental materialization keeps columns whose builder inputs are unchanged, so an extension could compare its outputs against full regeneration when prompts or models drift.","Because TableStrings encode row-selection patterns, TableVault could act as a plan substrate for cross-pipeline optimization, letting a planner rewrite a downstream builder's input range instead of regenerating an upstream table.","The versioned-instance model gives a concrete place to detect model or data drift: diffing instances built from the same builder at different timestamps reveals when an LLM's behavior or the underlying data has changed, which the paper leaves implicit."],"forward_implications":["A long-running instance generation can be stopped early or restarted from the last disk write, so a bad prompt or failed API call does not force a full rerun of the dataframe.","Because builders record parameters and generation timestamps, any instance can be regenerated from its YAML recipe whenever the external environment is preserved, supporting audits and error recovery.","TableString references make provenance explicit: every column that depends on another table instance carries a recorded dependency, so downstream queries can trace which inputs produced a result.","Table and instance-level locking lets a single-threaded notebook application run background operations concurrently without corrupting table state.","Incremental materialization reuses unchanged columns across instances, which cuts the number of expensive LLM calls when only part of a prompt changes."],"supporting_citations":[{"why":"Supplies the general transaction-recovery principles behind write-ahead logging and rollback that underpin the claimed durability.","marker":"[9]"},{"why":"Provides the write-ahead logging and partial-rollback recovery method that TableVault cites for its ACID behavior.","marker":"[15]"},{"why":"Supplies the concurrency-control background for coordinated locking in interactive systems, cited as the basis for isolation.","marker":"[8]"},{"why":"Exemplifies declarative LLM-pipeline systems with user parameterization, which TableVault's builders generalize.","marker":"[12]"},{"why":"Motivates the dataframe transformation patterns (reduce, one-to-one, aggregation, convolution, selection) that TableString encodes.","marker":"[21]"},{"why":"Prior work by the authors on dataframe lineage that TableVault extends to fine-grained provenance tracking.","marker":"[33]"}],"fun_headline_variants":["TableVault brings ACID transactions to LLM-driven data workflows","Versioned, lock-protected tables for concurrent LLM pipelines","TableVault: reproducible data management for AI workflows","Transactional table instances for composable LLM operations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the system actually implements what it describes—two-phase locking, write-ahead logs, rollbacks, and incremental materialization—and that these mechanisms deliver the stated ACID guarantees, since the paper does not show code, traces, or tests.","fun_headline_variants_meta":{"raw":{"variants":["TableVault brings ACID transactions to LLM-driven data workflows","Versioned, lock-protected tables for concurrent LLM pipelines","TableVault: reproducible data management for AI workflows","Transactional table instances for composable LLM operations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1408,"prompt_tokens":811,"completion_tokens":597,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":530}},"tokens_in":427,"tokens_out":597,"duration_ms":7001,"temperature":1.0,"reasoning_tokens":530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:21:52.956083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Kill TableVault mid-generation after some rows are written, restart it, and check whether the partially written instance is hidden and the last committed state is restored; if partial rows are visible, locks stay held, or the operation cannot resume under its original identifier, the ACID claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the general transaction-recovery principles behind write-ahead logging and rollback that underpin the claimed durability."},{"cited_title":"Mohan, Don Haderle, Bruce Lindsay, Hamid Pirahesh, and Peter Schwarz","cited_arxiv_id":null,"evidence_quote":"Provides the write-ahead logging and partial-rollback recovery method that TableVault cites for its ACID behavior."},{"cited_title":"Ellis and Simon J","cited_arxiv_id":null,"evidence_quote":"Supplies the concurrency-control background for coordinated locking in interactive systems, cited as the basis for isolation."}],"review_version":1}