{"id":"d7998b13-cb98-4320-8135-fe0a583a4cdd","arxiv_id":"2506.23719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"DABstep releases 450+ real-world financial analytics tasks requiring iterative code and documentation reasoning; the best baseline solves 14.55% of hard tasks.","lead":"DABstep is a new benchmark of over 450 real-world financial data analysis tasks for AI agents, scored by exact short answers. The best tested agent, o4-mini, solves only 14.55% of the hard tasks, exposing a large gap in multi-step reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 2.1's claim that no Hard task can be answered by a single direct execution is the load-bearing novelty premise, yet no single-shot baseline is reported and the term is undefined; the 14.55% result may reflect one-shot code-generation difficulty rather than required multi-step reasoning.","rationale":"The reader's conditional verdict already identifies the multi-step necessity claim as the weakest assumption; my analysis agrees and sharpens it. Section 2.1's 'no question can be answered through a single direct execution' is the load-bearing premise because it separates DABstep from single-shot code/SQL benchmarks, and because the headline finding (14.55% on Hard) is only interpretable as a multi-step reasoning gap if that premise holds. The paper neither defines the term operationally nor reports a single-shot baseline; all models in Table 1 use a ReAct loop with up to 10 steps. The A.4 trace demonstrates iterative behavior for one task but does not prove that a single comprehensive script would fail. Without a control, the 14.55% figure could equally reflect the difficulty of producing a correct script in one shot, which would make DABstep closer to DS-1000 or Spider than to a genuine multi-step reasoning benchmark. I considered the A.2 numeric-tolerance inconsistency (10^-4 in text vs 10^-2 in Algorithm 1) and the absence of error bars, but these affect score precision and model ranking, not the benchmark's core identity; they are fixable in revision. The single-shot control is the decisive missing experiment. Because the reader's conditional verdict already flags this and asks for evidence, my read does not change the verdict; it strengthens the condition.","tokens_in":13617,"tokens_out":8072,"duration_ms":90883,"concrete_test":"Run a single-shot control on a random sample of at least 50 Hard tasks from the developer set. For each task, prompt a strong model (e.g., o4-mini or Claude 3.7 Sonnet) with the task text, guidance, and context filenames, and allow it to emit exactly one Python script; execute that script once with no further feedback, and score with the same hybrid scorer. Compare this to the multi-step ReAct baseline on the same sample. If single-shot accuracy is within a few points of the ReAct accuracy, the 'cannot be answered through a single direct execution' claim is falsified, and the benchmark should be reframed as measuring feedback-driven exploration rather than task-inherent multi-step reasoning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 makes the load-bearing claim that 'in the Hard split (84% of tasks), no question can be answered through a single direct execution. Hard tasks require iterative data filtering and cross-referencing, which single-shot code cannot handle.' This premise distinguishes DABstep from Text-to-SQL and code-generation benchmarks and underwrites the interpretation of Table 1's 14.55% Hard accuracy as a multi-step reasoning deficit. The premise is not established. No single-shot control is reported: all baselines in Section 4.1 use a ReAct loop with a maximum of 10 steps. Moreover, 'single direct execution' is ambiguous. If it means 'one Python script,' the claim is unlikely to hold in general, because a script can load payments.csv, fees.json, and manual.md, then join, filter, and apply rules in a single run; what forces iteration is the agent's need to discover schemas and rules, which is a property of the agent's information state, not of the task's computational structure. If it means 'one LLM call without execution feedback,' that is a harness choice, and the paper provides no measurement of it. The A.4 trace illustrates one 7-step trajectory but does not show that a single comprehensive script would fail. Thus the core novelty claim is asserted rather than demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"DABstep introduces a benchmark of over 450 financial data analysis tasks derived from Adyen's operational workloads, combining structured datasets (CSV, JSON) with unstructured documentation (Markdown). Tasks have factoid answers and are scored with a deterministic hybrid algorithm that was validated against human judgments on 75 answers (100% agreement). The paper evaluates 16 LLM-based agents under a ReAct-style harness with up to 10 steps, reporting that the best model (o4-mini) achieves only 14.55% accuracy on the Hard split (378 tasks) versus 76.39% on Easy tasks. The benchmark is released with a public leaderboard, open-source baselines, and a developer set. The central claim is that Hard tasks require genuine multi-step reasoning that distinguishes DABstep from Text-to-SQL and code-generation benchmarks.","tokens_in":13914,"tokens_out":5341,"duration_ms":53140,"significance":"If validated, DABstep would be a valuable resource: it provides real-world, heterogeneous data analysis tasks with objective, scalable evaluation, and the low hard-split accuracy suggests a meaningful gap in current agent capabilities. The deterministic scoring protocol, the human validation of that protocol (100% agreement on 75 examples), the public leaderboard, and the release of code and data are concrete strengths that support reproducibility and community use. The paper's novelty, however, rests on the assertion that Hard tasks cannot be solved by one-shot code generation; this is not yet demonstrated, and the lack of a single-shot control weakens the interpretation of the headline 14.55% result as a multi-step reasoning deficit. A human expert baseline would also help calibrate task solvability. These issues are fixable and do not undermine the overall enterprise, but they need to be addressed before the benchmark's distinctive contribution is fully established.","major_comments":[{"comment":"The load-bearing claim that Hard tasks 'cannot be answered through a single direct execution' is not supported by the evidence. All baselines in Section 4.1 use a ReAct loop with a maximum of 10 steps and execution feedback; no single-shot code generation condition is reported. The term 'single direct execution' is ambiguous: if it means one Python script, then a script could in principle load all context files (payments.csv, fees.json, manual.md) and perform the required joins, filters, and rule-based computations in a single run, so what forces iteration is the agent's need to discover schemas and rules rather than the computational structure of the task. If it means one LLM call without execution feedback, that is a harness choice that the paper does not measure. The Appendix A.4 trace shows a failed 7-step trajectory but does not demonstrate that a single comprehensive script would fail. Without a one-shot control, the interpretation of Table 1's 14.55% Hard accuracy as a multi-step reasoning deficit is not established; it could reflect one-shot code generation difficulty alone. Please add a single-shot baseline (e.g., prompt the model to write a complete script and execute it without iterative feedback) and, if the claim is retained, provide a formal definition of what constitutes a required 'step' and verify per task that the stated minimum step count is necessary.","section":"Section 2.1, 3.2, and 4.1"},{"comment":"No human expert baseline is reported. The paper states that tasks are derived from real internal queries, which suggests solvability, but it does not demonstrate that the provided context files are sufficient for a human to answer each question as posed. Without a human performance estimate on a sample of tasks, the low agent accuracies could in principle reflect task ambiguity or underspecification rather than limitations in agent reasoning. The validation in Appendix A.2 (75 answers, 100% scoring agreement) checks only the scoring function, not the solvability or clarity of the tasks themselves. I recommend reporting human expert accuracy on a representative sample (for instance, from the developer set) and, if any tasks are found to be ambiguous or unsolvable from the given context, either revising those tasks or documenting the judgment criteria.","section":"Section 5 and A.2"}],"minor_comments":[{"comment":"The phrase 'gains in: high evaluation reliability' is awkwardly constructed; consider rewording to 'gains in evaluation reliability, scalability...'.","section":"Section 2.2"},{"comment":"The caption states that all baselines run for a maximum of 10 steps with a ReAct-style prompt 'except for the reasoning models'; please clarify whether reasoning models also have a 10-step limit or a different limit, since this affects the comparability of results.","section":"Table 1 caption"},{"comment":"The text specifies a numeric tolerance of 10^-4, but Algorithm 1 uses tolerance=10^-2; please reconcile this discrepancy.","section":"Appendix A.2, Algorithm 1"},{"comment":"The sentence 'From our baselines in Section 4, there is a 49% correlation with performance on the easy set to performance on the hard set' is vague; please specify which correlation coefficient is used and whether it is computed across models or across tasks.","section":"Section 3.2"},{"comment":"The figure caption refers to 'at least 3 execution steps' and 'at least 6 steps' for Easy and Hard tasks; please define what constitutes an execution step and whether this is the number of agent actions or code executions.","section":"Figure 1"},{"comment":"The discussion of self-attention and abstract conceptual similarity is speculative; consider softening the claim or providing citations to empirical evidence that directly supports this explanation.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is written by authors affiliated with Adyen and Hugging Face, and it cites the authors' own smolagents library and the co-authored GAIA benchmark; this is not inherently problematic, but the report should note that the benchmark is drawn from a single company's domain (financial payments), which may limit its generalizability beyond that vertical. The most consequential issue is the lack of a single-shot control, which directly affects the interpretation of the headline result. If the authors add such a control and a human baseline, the paper could become a solid contribution; without these, the novelty claim remains unsubstantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: DABstep is a genuinely useful addition to the agent-evaluation zoo. The tasks come from real Adyen workflows, the data are released, the scoring is deterministic and was checked against human judgments on 75 answers with 100% agreement, and the hidden test set is a sensible way to slow saturation. The baseline numbers are plausible, and the low Hard accuracy is meaningful.\n\nThe main soft spot is the one the stress-test flags. Section 2.1 says no Hard task can be answered by a single direct execution, and that claim carries the interpretation of 14.55% as a multi-step reasoning deficit. But there is no single-shot baseline. All reported agents run a ReAct loop with up to 10 steps. Without a control that gives the model one code execution attempt (or one full script) per task, we don't know how much of the gap is iterative planning versus simply writing one complex program. The term \"single direct execution\" is ambiguous—a single Python script could load all files, join, filter, and compute in one go. What forces iteration is the agent's uncertainty about schemas and rules, which is about the information state, not the computational requirement. That doesn't sink the benchmark, but it means the novelty claim is overstated.\n\nTwo smaller issues. No human expert baseline, so we can't calibrate whether 14.55% is 'hard' or 'models are bad.' And the scoring tolerance is inconsistent: the text says 10^-4, the algorithm says 10^-2. Minor, but fixable. Also, the 95 core questions expanded to 450 variants is fine, but it means effective diversity is lower than the raw count suggests.\n\nThe citation pattern is fine. Self-citing smolagents and GAIA is not a problem; the baselines use the authors' own code, but the benchmark itself is released openly. No circularity burden.\n\nWho is this for? Anyone building or evaluating data-analysis agents. It's a legitimate benchmark with real tasks and a working leaderboard. It deserves peer review, but the single-shot control and a human baseline should be required before acceptance.\n\nMy vote: would send to reviewers, with the expectation of a revision. We'd probably cite it if we do agent evaluation work.","headline":"DABstep is a solid, usable benchmark for data-analysis agents, but the central claim that Hard tasks are impossible single-shot is asserted rather than demonstrated, and the baselines would be much stronger with a single-shot control and error bars.","tokens_in":14439,"tokens_out":1952,"would_cite":true,"duration_ms":20761,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DABstep is a new benchmark of 450+ real financial-analysis tasks, and it shows that even the best LLM agent solves only 14.55% of the hard ones.","keywords":["DABstep","multi-step reasoning","LLM agents","data analysis benchmark","financial analytics","factoid evaluation","code execution","heterogeneous data"],"falsifier":"Run an expert analyst or a single-pass code-generation system on the 378 Hard tasks with no iterative feedback; if a substantial fraction are solved correctly in one execution, the benchmark's defining multi-step requirement would not hold as stated.","tokens_in":13456,"feed_emoji":"📊","tokens_out":4849,"duration_ms":47582,"temperature":0.7,"pith_summary":"DABstep is a benchmark of more than 450 data-analysis tasks drawn from real financial-analytics workloads. Each task forces an agent to combine code execution, heterogeneous data files, and domain documentation, and to answer in a machine-checkable factoid format. The paper's central claim is that these tasks require genuine multi-step reasoning and that current LLM agents are far from doing that well: the best baseline solves only 14.55% of the hard split, while easy single-step tasks are largely mastered. If this holds, the benchmark identifies multi-step planning and documentation-grounded reasoning, rather than basic code or SQL generation, as the current bottleneck in autonomous data analysis.","feed_headline":"Best AI agents solve only 14.55% of hard data-analysis tasks","feed_subtitle":"A benchmark of 450+ real financial-analysis tasks shows multi-step reasoning, not code, is the current limit.","key_machinery":"The load-bearing object is the benchmark task structure itself: each item pairs a natural-language question with heterogeneous context files, formatting guidance, and a difficulty tag. Its main design mechanism is the parameterization of 95 core workflows into over 450 instances, so success cannot come from memorized answers. The factoid answer format plus a hybrid scoring algorithm that tolerates numeric, list, and string variations provides objective, LLM-free evaluation, and a hidden test set keeps the measure focused on zero-shot generalization.","core_discovery":"The paper introduces DABstep and argues that state-of-the-art LLM agents reliably handle single-shot analyses but largely fail when analysis requires iterative decomposition. Of 450+ tasks, 378 are classified Hard and designed so no question can be answered through a single direct execution; agents must filter, aggregate, consult manuals, compute intermediate results, and refine. On a hidden test set with objective factoid scoring, the best agent reaches 76.39% on Easy but only 14.55% on Hard, and every model tested drops sharply on the Hard split. The paper treats this gap as evidence that multi-step reasoning over heterogeneous data sources is an unsolved capability, and it releases the tasks, scoring code, baselines, and a leaderboard to make that gap measurable.","pith_inferences":["A testable extension would be to check whether increasing the number of allowed execution steps or giving agents an explicit planning scratchpad raises Hard accuracy substantially; the paper's standardized loop caps each task at ten steps, which may understate model capability.","If the single-execution premise is wrong for many Hard tasks, part of the measured gap may be an artifact of scoring or prompting rather than of multi-step reasoning itself.","The parameterization design suggests a natural stress test: permuting the same 95 core tasks further could expose whether models generalize the reasoning procedure or overfit to surface patterns."],"forward_implications":["If DABstep's Hard split is taken as representative, current agents cannot be trusted for unsupervised real-world financial data analysis.","The score gap between Easy and Hard splits indicates that single-step code or SQL proficiency is no longer the limiting factor; decomposition and planning are.","The release of a public leaderboard and developer set makes agent performance on these tasks repeatable and comparable without heavy scaffolding.","Low-cost open models score close to zero on Hard, so cost-performance trade-offs and model capability are strongly coupled for iterative analysis.","Failure analysis implies that improvements in instruction following and documentation-aware planning may matter more than raw coding ability."],"supporting_citations":[{"why":"Supplies the top baseline result that defines the reported capability ceiling of 14.55% on Hard tasks.","marker":"[34]"},{"why":"Supplies the factoid hidden-test evaluation template and the design precedent for a held-out benchmark set.","marker":"[28]"},{"why":"Supplies the parameterization approach that expands 95 core questions into 450+ instances.","marker":"[29]"},{"why":"Supplies the code-generation benchmark comparison that DABstep argues does not require multi-step reasoning.","marker":"[7]"},{"why":"Supplies the Text-to-SQL baseline paradigm that DABstep distinguishes itself from.","marker":"[47]"},{"why":"Supplies the comparison to enterprise multi-step SQL workflows and motivates heterogeneous data integration.","marker":"[23]"},{"why":"Supplies the reasoning-action loop used to run the baseline agents.","marker":"[45]"},{"why":"Supplies the argument for standardized baseline evaluation that motivates the hidden test set and leaderboard.","marker":"[35]"}],"fun_headline_variants":["AI agents stumble at multi-step data analysis: 14.55% accuracy","DABstep benchmark: 450+ tasks, best AI scores 14.55% on hard ones","Even top AI agents fail complex data reasoning, new benchmark shows","Multi-step reasoning is AI's Achilles' heel, DABstep reveals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that every Hard task truly demands multiple sequential reasoning steps, and the paper does not demonstrate, formally or empirically, that a single well-crafted script cannot answer one of them.","fun_headline_variants_meta":{"raw":{"variants":["AI agents stumble at multi-step data analysis: 14.55% accuracy","DABstep benchmark: 450+ tasks, best AI scores 14.55% on hard ones","Even top AI agents fail complex data reasoning, new benchmark shows","Multi-step reasoning is AI's Achilles' heel, DABstep reveals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2608,"prompt_tokens":847,"completion_tokens":1761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":463,"tokens_out":1761,"duration_ms":11872,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:33:09.500207+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an expert analyst or a single-pass code-generation system on the 378 Hard tasks with no iterative feedback; if a substantial fraction are solved correctly in one execution, the benchmark's defining multi-step requirement would not hold as stated.","supporting_citations":[{"cited_title":"Introducing openai o3 and o4-mini","cited_arxiv_id":null,"evidence_quote":"Supplies the top baseline result that defines the reported capability ceiling of 14.55% on Hard tasks."},{"cited_title":"Gaia: a benchmark for general ai assistants","cited_arxiv_id":null,"evidence_quote":"Supplies the factoid hidden-test evaluation template and the design precedent for a held-out benchmark set."},{"cited_title":"GSM-symbolic: Understanding the limitations of mathematical reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the parameterization approach that expands 95 core questions into 450+ instances."},{"cited_title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task","cited_arxiv_id":null,"evidence_quote":"Supplies the Text-to-SQL baseline paradigm that DABstep distinguishes itself from."},{"cited_title":"Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows","cited_arxiv_id":null,"evidence_quote":"Supplies the comparison to enterprise multi-step SQL workflows and motivates heterogeneous data integration."}],"review_version":1}