{"id":"ccc4d14f-1b14-4345-833b-6cc52ca47f5a","arxiv_id":"2608.07873","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The workbook time machine reverse-engineers real workbooks into creation tasks, and the resulting benchmark shows pivot tables remain nearly unsolved by all tested frontier models.","lead":"The authors build a 'workbook time machine' that strips finished Excel files back to raw data and generates natural-language tasks asking AI models to recreate the removed formulas, charts, pivot tables, and formatting rules. They publish 150 such tasks as WTM-BENCH and show that API choice, query detail, and agent design change success rates more than model size does.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's validity depends on the backward step removing Excel's cached formula values; if any cached <v> survives into the raw input, tasks leak their answers and the reported model/API ordering does not measure spreadsheet creation.","rationale":"The reader's conditional verdict is appropriate, and the identified weakest assumption is the same one I would flag. The central claim requires WTM-BENCH to measure creation of derived artifacts; if input workbooks contain cached values or descriptor labels, models can copy rather than compute. The paper provides no explicit handling of cached formula values and no audit that would detect their presence. The Future Work section explicitly lists 'formula descriptor detection' as a way to 'minimize structural information leakage', indicating that the stripping step is acknowledged to be incomplete. The proposed test is inexpensive and would settle the concern: if input files contain no cached values in stripped formula ranges and the no-op baseline scores zero on formula tasks, the benchmark's validity on this axis is supported; if the test fails, the reported model ordering and all task-level conclusions need re-interpretation. Other concerns, such as the 'exact reconstruction' framing versus the lenient grader, are secondary to this leakage risk because they affect interpretation of scores rather than the task content itself. Therefore I do not move the verdict; it remains conditional pending this check.","tokens_in":17297,"tokens_out":4692,"duration_ms":56207,"concrete_test":"Run a leakage audit on the released WTM-BENCH inputs. (1) Select all ~62 formula tasks; open each input workbook with openpyxl in data_only=True mode and scan the cells in ranges that the ground truth fills with formulas; record whether any non-empty numeric values are present where the raw state should be empty. (2) Run a no-op baseline: for each task, return the input workbook unchanged as the model output and compute Soft/Hard scores. If any formula task has cached values in the input, or if the no-op baseline scores >0 on formula tasks, the stripping step leaks answers and the benchmark's validity and reported ordering are not established. If both checks are clean, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the backward stripping in §4.1. For every triple (Win, Wout, q), Win must be a clean raw state from which Wout is genuinely derivable. The paper never states how Excel's cached formula values are handled. An xlsx formula cell stores its last computed value in <v>; if the pipeline removes the formula element but leaves that cached value, or if stripping is done via a data_only read that preserves values, then Win already contains the answer to a formula task. The model can succeed by copying or even doing nothing, and the grader in §H credits cell values computed via formula results, so a no-op would receive full formula credit. This would invalidate every formula task and distort the artifact-type and complexity analyses. The risk is not hypothetical: §4.1's 'semantic descriptor mapping' is acknowledged as incomplete in the Future Work section, where 'formula descriptor detection' is listed as future work to 'minimize structural information leakage', and Appendix A admits that manually entered lookup values are not handled. No section describes cached-value removal, leakage testing, or a control baseline that would detect it. Since all 150 tasks and the headline API/query/orchestration ordering depend on these triples, this is the weakest load-bearing link.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes the workbook time machine, a generation pipeline that starts from real user-authored Excel workbooks, removes derived artifacts (formulas, charts, pivot tables, conditional formatting) to recover a raw input state, reconstructs candidate edit histories as a dependency-pruned DAG, and generates natural-language queries at three specificity levels. From Enron and FUSE it produces WTM-CORPUS (8,931 queries over 2,977 tasks) and a curated WTM-BENCH (150 tasks, 450 queries). The paper evaluates six frontier LLMs under three Excel APIs and three orchestration frameworks, reporting that API choice, orchestration, and query specificity explain more variance than model identity, with pivot tables remaining near-unsolved.","tokens_in":17601,"tokens_out":8012,"duration_ms":90498,"significance":"WTM-BENCH fills a real gap: it is the only benchmark I know of that combines realistic workbooks, multi-artifact creation, and controlled instruction specificity. The design has genuine strengths: the grading is deterministic and programmatic with no LLM judge; the three query levels provide a useful diagnostic; the artifact-type balance is deliberate; and the independent Python re-runs in Appendix G show stable model ordering. The main claims are therefore meaningful if the benchmark triples are leakage-free. That conditional is not yet established, and it is the central point on which the paper's validity rests.","major_comments":[{"comment":"The backward step's treatment of cached formula values is not described. In the xlsx format, a formula cell stores both the formula and its last computed value in <v>; if the backward step does not remove those cached values, then Win already contains the numeric answer to formula and downstream artifact tasks. The only trace is the unexplained 'Cached - out' label in Figure 2, and Appendix H grades cell values by computing formula results, so a model that copies the cached values (or does nothing) can receive full formula credit. The Future Work section admits that formula descriptor detection is still open, and Appendix A admits that manually entered lookup values are not handled, which are related leakage channels. Please specify exactly how cached values are stripped, verify on the released corpus that no <v> survives in Win, and report a no-op/copy control baseline and a leakage test. Because all 150 tasks and the headline API/query/orchestration ordering depend on these triples, this is load-bearing.","section":"§4.1 and Figure 2"},{"comment":"The Edit DAG pruning and the fallback rule (retain the connected artifact containing W when the pruned graph is disconnected) are justified by assertion rather than validation. No experiment checks that each retained triple has at least one valid edit sequence from Win to Wout, or that the LLM-based semantic descriptor mapping and necessity-score annotations are correct. The human study in Appendix B rates query naturalness and completeness, not task derivability. I would like to see a derivability audit on a sample (for example, executing the intended action parameters from Win and comparing the result with Wout), or at least a human-validated subset with agreement statistics.","section":"§4.2 and Appendix B"}],"minor_comments":[{"comment":"Hard Score is described as 'exact task reconstruction', but Soft Score intentionally ignores axis labels, colors, rule priority, and aggregation function names, so Hard Score is exact only with respect to the weighted component metrics, not exact workbook reconstruction.","section":"§8 and Appendix H"},{"comment":"The labels 'lvl1, lvl3, lvl5' appear even though the paper defines three query levels; this is a typo that should be corrected.","section":"Figure 16(b)"},{"comment":"The phrase 'sequential generation (Level 1→2→3) sequentially in 1 LLM call' is confusing; please clarify whether the three levels are produced in a single call or in sequential calls, since the two interpretations have different cost and leakage implications.","section":"§5 and Appendix C"},{"comment":"Re-run robustness is reported only for Python/OpenPyXL; the paper should state explicitly that the VBA and OfficeJS results are single-run, or provide re-runs for those APIs if the API-comparison claim is to carry equal weight.","section":"Appendix G"},{"comment":"The comparison across APIs is confounded by backend-specific system prompts and function-calling schemas; this should be acknowledged as a limitation of the API-effect interpretation.","section":"§7 and Appendix F"},{"comment":"The same model (GPT-5.4 Reasoning) is used both to generate the benchmark queries and as one of the evaluated models; the paper should discuss whether this creates any style or format advantage and whether contamination was considered.","section":"§5 and §7"}],"recommendation":"major_revision","confidential_remarks":"The cached-value issue is the key gate for this paper. If the authors can demonstrate that cached formula values are stripped and provide a leakage control, the benchmark construction is credible and the paper would be publishable. I would not accept without that evidence. I also recommend that the authors double-check the novelty claim against SpreadsheetBench and SheetCopilot, and add a sentence on potential contamination from using the same model family for generation and evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is the strongest spreadsheet-creation benchmark paper I've read in a while, and it has a leak-shaped hole that needs closing before the scores can be taken at face value.\n\nThe workbook time machine is a genuine contribution. Reverse-engineering finished workbooks into (input, output, query) triples via dependency-aware DAGs, then generating queries at three specificity levels, is a sensible way to get ecologically valid tasks with controlled variation. WTM-BENCH is the first benchmark I know that jointly covers realistic workbooks, multi-step creation of formulas, charts, pivot tables, and conditional formatting, and instruction granularity in one instrument. The evaluation is careful in the right places: deterministic programmatic grader, no LLM judge, independent re-runs reported for the Python configuration with standard deviations, and a failure taxonomy that separates wrong shape, wrong placement, and wrong values. The headline findings — that API choice and agent orchestration matter more than model identity, that pivot tables are nearly unsolved, and that performance degrades with task complexity and query abstraction — are plausible and useful.\n\nThe soft spot is the backward stripping step, and the stress-test note targets the right question. The paper never states how cached formula values are handled. An xlsx formula cell stores its last computed value in the file; if the pipeline removes the formula element but leaves that cached value, or strips via a data_only read, then the input workbook already contains the answer to every formula task. The grader credits cell values computed via formula results and compares only changes from the pre-task baseline, so a model that does nothing would receive full formula credit. That would invalidate every formula task and distort the artifact-type and complexity analyses. The paper's own Future Work admits \"formula descriptor detection\" is needed to \"minimize structural information leakage,\" which is a direct acknowledgment that current stripping is incomplete. Nothing in the main text or appendices describes removing cached values, testing for leakage, or running a no-op control baseline.\n\nI don't think this is fatal to the idea, but it is load-bearing under-specification. The fix is straightforward: describe the exact stripping mechanics and run a control showing that a no-op agent scores zero on formula tasks. Until that is done, the reported ordering should be treated as provisional. A related, minor issue: the Hard Score is called \"exact task reconstruction,\" but it is really \"perfect Soft Score,\" which is a grader-relative notion; the phrasing is stronger than the metric supports.\n\nThis paper is for anyone building or evaluating spreadsheet agents, and for people working on reverse-curriculum benchmark construction. It deserves a serious referee — I would send it to review, but with a clear request to resolve the leakage question and to re-report the scores if any leak is found. Engage with it, but don't trust the benchmark as a diagnostic until the cached-value issue is closed.","headline":"The WTM pipeline is a real step forward for spreadsheet-creation benchmarks, but the unstated handling of Excel's cached formula values is a load-bearing caveat; the paper deserves review, not blind trust.","tokens_in":18087,"tokens_out":6261,"would_cite":false,"duration_ms":61369,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pipeline that erases derived artifacts from real Excel workbooks yields a 150-task benchmark, and evaluation on it shows that API choice, query specificity, and agent orchestration dominate model identity in spreadsheet-creation…","keywords":["workbook time machine","spreadsheet benchmark","LLM agents","formula generation","pivot tables","conditional formatting","instruction specificity","reverse curriculum"],"falsifier":"Take a real workbook in which a formula cell contains its last cached computed value, run the workbook time machine's backward step, and inspect the recovered raw workbook: if any numeric cell remains at the stripped formula's position, a model could copy that value instead of deriving it, and every score on the benchmark built from that triple is suspect.","tokens_in":17134,"feed_emoji":"📊","tokens_out":6979,"duration_ms":65538,"temperature":0.7,"pith_summary":"This paper presents the workbook time machine, a pipeline that turns finished, user-authored Excel workbooks into benchmark triples: an input workbook, an output workbook, and a natural-language query. The defining move is to run backward: strip formulas, charts, pivot tables, and conditional-formatting rules from a real workbook to recover the raw data, then run forward to generate instructions at controlled levels of specificity. Applied to public spreadsheet corpora, it produces a corpus of 8,931 queries and a curated 150-task benchmark balanced across artifact types. Evaluations on that benchmark claim that spreadsheet agents succeed or fail mainly because of the scripting API used to control Excel, how detailed the instruction is, and how the agent loop is orchestrated, rather than which frontier model is chosen. If correct, this gives the field a diagnostic instrument that can separate pure execution failures from instruction-following failures.","feed_headline":"API and instruction detail outweigh model choice in spreadsheet agents","feed_subtitle":"A reverse pipeline turns real Excel workbooks into 150 creation tasks; pivot tables stump every tested model.","key_machinery":"The load-bearing mechanism is the Edit DAG, a directed acyclic graph whose nodes are workbook states obtained by adding subsets of the derived artifacts to the raw workbook and whose edges add one artifact at a time. Dependency-aware pruning removes edges whose prerequisite artifacts are absent, so only semantically valid edit histories remain; sub-paths of the pruned DAG become benchmark triples. A companion necessity score labels each action parameter as essential, useful, or incidental, and an LLM turns filtered parameter sets into natural-language instructions at three specificity levels. The DAG also yields a task-complexity measure: the number of artifact transformations between input and output states.","core_discovery":"The paper's central claim is that WTM-BENCH is the first spreadsheet benchmark to jointly test language models on realistic, user-authored workbooks; on multi-step creation of formulas, charts, pivot tables, and conditional formatting; and on instructions whose specificity is controlled. The argument runs through the workbook time machine: starting from a finished workbook, it removes derived artifacts to recover a raw state, builds a dependency-pruned edit DAG of plausible artifact-creation histories, and generates natural-language queries at three specificity levels for each transformation. Evaluation across eighteen model-API combinations and several orchestration frameworks shows that query specificity, agent orchestration, and the Excel interface API shape performance more than model identity, with the best configuration using a native macro language. Formulas are the most tractable artifact, pivot tables remain nearly unsolved, and performance degrades monotonically as instructions become more abstract and as tasks require more steps.","pith_inferences":["If cached formula values are not fully stripped by the backward step, a model could copy the stored result instead of deriving it; a natural stress test is to run the pipeline on workbooks with stale cached values and measure the score drop.","The pipeline's direction can be inverted to create deletion and modification tasks by swapping input and output states—an extension the paper lists as future work but does not itself evaluate.","The finding that a large share of failed rollouts end with a confident completion claim suggests that practical spreadsheet agents need an independent verification step, not just better instruction-following.","If the specificity gradient holds up, training curricula that start at the most detailed query level and move to abstract queries could improve robustness to ambiguous instructions, but this remains to be tested."],"forward_implications":["WTM-BENCH scores should be read as properties of the whole stack—model, API, query level, and orchestration loop—not of model capability alone.","Comparing models through a single API can change their relative ranking, so benchmark results should report API as a condition rather than a confound.","Pivot-table creation is an open problem at the current frontier; a score below ten percent soft accuracy marks a clear target for improvement.","The three query levels give a direct way to tell whether a model fails because it cannot follow a terse instruction or because it cannot execute the underlying transformation, since the same workbook transformation is described at all three specificities."],"supporting_citations":[{"why":"Supplies the Enron workbook corpus, the source of real user-authored files for benchmark triples.","marker":"(Hermans & Murphy-Hill, 2015)"},{"why":"Supplies the FUSE workbook corpus, the second source of real user workbooks.","marker":"(Barik et al., 2015)"},{"why":"Motivates formula grouping, the heuristic that collapses repeated drag-filled formulas into single artifacts.","marker":"(Gulwani, 2011)"},{"why":"Supplies the reverse-curriculum idea of constructing benchmarks by working backward from goal states.","marker":"(Florensa et al., 2017)"},{"why":"Supplies hindsight-style reasoning that candidate edit histories can be reconstructed from final states.","marker":"(Andrychowicz et al., 2017)"},{"why":"Provides cell-dependency analysis that the paper's dependency-aware DAG pruning builds on.","marker":"(Zhu et al., 2025)"},{"why":"Supplies SheetCopilot, the prior benchmark with diverse artifact types on hand-crafted workbooks that WTM-BENCH extends to real files.","marker":"(Li et al., 2023)"},{"why":"Supplies SpreadsheetBench, the realistic-workbook formula benchmark used as a comparison baseline in evaluation.","marker":"(Ma et al., 2024)"}],"fun_headline_variants":["Spreadsheet AI: API and query precision beat model size","Reverse time machine turns Excel files into AI benchmarks","Pivot tables stump every LLM in new 150-task test","Agent orchestration trumps model choice for Excel AI","Benchmark: API and instructions matter more than model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the backward step can strip formulas, charts, pivot tables, and conditional formatting from a finished workbook and recover a clean raw input without leaking the answer—in particular, that cached formula values and adjacent descriptor cells do not survive the stripping.","fun_headline_variants_meta":{"raw":{"variants":["Spreadsheet AI: API and query precision beat model size","Reverse time machine turns Excel files into AI benchmarks","Pivot tables stump every LLM in new 150-task test","Agent orchestration trumps model choice for Excel AI","Benchmark: API and instructions matter more than model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2919,"prompt_tokens":847,"completion_tokens":2072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1993}},"tokens_in":463,"tokens_out":2072,"duration_ms":15378,"temperature":1.0,"reasoning_tokens":1993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:44:24.016029+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real workbook in which a formula cell contains its last cached computed value, run the workbook time machine's backward step, and inspect the recovered raw workbook: if any numeric cell remains at the stripped formula's position, a model could copy that value instead of deriving it, and every score on the benchmark built from that triple is suspect.","supporting_citations":[],"review_version":1}