{"id":"41c237b0-8127-4658-b3bb-3a13b4b82a91","arxiv_id":"2507.23336","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new 303-question benchmark of realistic data science tasks finds that Claude-4-Sonnet with multi-step code generation outperforms other LLMs and agent styles, but the accuracy numbers rest on a thinly validated automatic judge.","lead":"This paper introduces DSBC, a new benchmark of 303 realistic data science questions for evaluating LLM-based data science agents. It compares three LLMs and three agent approaches across 165 settings each, reporting on accuracy, temperature sensitivity, and query wording.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy comparison is only as trustworthy as a single unvalidated VLM judge; with only 3 manual checks and small accuracy gaps, correlated judge errors could flip the reported ranking.","rationale":"The strongest claim is empirical: the benchmark reveals that Claude-4-Sonnet with multi-step code generation clearly outperforms the other models and approaches. For this claim to hold, the accuracy measurements have to be valid. The paper's only evidence for validity is three manual checks of the Gemini-2.5-Flash judge (Section 7). Three observations cannot estimate a mislabel rate, let alone detect differential bias across the 15 configurations compared in Table 7. The margins separating the top configurations are only a few percentage points, so even a modest, treatment-correlated judge error could change the ranking. This is especially concerning because the judge is the same model family as one of the evaluated systems (Gemini-2.5-Flash), and because the judge prompt tells it to accept rounded numeric answers and 'Yes' when the response is 'technically correct'—criteria that leave room for inconsistent interpretation of code-generated outputs. The paper's own error analysis (Section 9) shows 70% of failures are 'incorrect final response,' which is exactly the category the judge must adjudicate, so the judge is not merely checking code execution but making semantic judgments. I also considered the inaccessibility of the dataset and code (Section 10 announces release but gives no link) and the SmolAgent step-limit inconsistency (8 vs. 5). These are real reproducibility issues, but they would be fixable after the fact and do not directly undermine the internal comparison if the evaluation were sound. The VLM-judge issue attacks the measured quantities themselves. The reader's CONDITIONAL verdict is appropriate; I would keep it CONDITIONAL. If the manual audit proposed above shows high agreement and preserves the ranking, the central claim would be substantially stronger. If the audit shows disagreement, the paper should either re-report with a human-validated subset, add a deterministic checker for numeric outputs, or present the results as exploratory rather than conclusive.","tokens_in":13207,"tokens_out":4957,"duration_ms":50222,"concrete_test":"Manually label a stratified random sample of at least 100 judge decisions, spanning all 3 LLMs, both query types, at least 3 temperature values, and all 8 task categories, using the stated ground-truth code and output as the reference. Compute human-judge agreement (Cohen's kappa) and a confusion matrix. Then recompute the overall accuracy for each of the 15 configurations in Table 7 using only the human labels instead of the VLM labels. If the top-ranked configuration changes, or if the Claude Multi-Code advantage over SmolAgent becomes smaller than the margin in Table 7, the paper's central claim should be weakened. As a cheaper complementary check, run the same judge prompt with two independent judge models (e.g., Claude-4-Sonnet and o4-mini) on a shared subset and measure pairwise agreement; low agreement would confirm that the single-judge design is unreliable.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim (Section 8) is that Claude-4-Sonnet with multi-step code generation clearly outperforms the other models and approaches. Every accuracy number in Table 7 and every temperature-sensitivity observation in Figures 4 and 46 comes from one VLM judge, Gemini-2.5-Flash, using the prompt in Appendix A (Prompt 5). Section 7 reports that only 3 of the 165 attempts were manually checked; since each of the 303 samples is evaluated over 165 attempts, this is at most 3 out of 49,995 judge decisions, and no inter-annotator agreement is reported. The load-bearing weakness is that the judge errors are not shown to be independent of the treatment variables. The model being judged includes Gemini-2.5-Flash itself, so self-preference or formatting biases could differentially favor one configuration. The headline margins are small: Claude Multi-Code Raw (59.92%) leads Claude SmolAgent Clean (55.24%) by about 4.7 points overall, and several adjacent configurations differ by only 2-3 points. A judge error rate of even 5%, if correlated with model or approach, is enough to change the ordering and would also alter the 'no significant temperature pattern' conclusion in Section 8. Three manual checks cannot detect systematic bias across models, task categories, or temperatures. A secondary reproducibility issue is the SmolAgent step-limit inconsistency (8 steps in Section 7.3 vs. 5 in Appendix B), but the judge validation is the more fundamental threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DSBC, a data science agent benchmark built from 303 queries derived from observed usage of the authors' commercial data science applications. The benchmark spans eight task categories, uses 11 Kaggle datasets, and applies a structured context-engineering format rather than free-form data descriptions. The authors evaluate three LLMs (Claude-4-Sonnet, Gemini-2.5-Flash, OpenAI-o4-Mini) across three approaches (single-step code, multi-step code, SmolAgent), two query types (raw and clean), and eleven temperature values, yielding 165 attempts per sample. Accuracy is determined by a Gemini-2.5-Flash VLM-as-a-judge with a JSON schema and regex parsing. The central finding is that Claude-4-Sonnet with multi-step code generation outperforms the other configurations, that multi-step and SmolAgent beat single-step approaches overall, and that no significant temperature pattern is observed. The benchmark and evaluation framework are released under Apache 2.0.","tokens_in":13547,"tokens_out":2587,"duration_ms":30381,"significance":"If the evaluation methodology is sound, the benchmark is a useful contribution: it is derived from real user interactions, includes multi-label task categories, pairs raw and clean versions of queries to study data leakage, and uses context engineering to avoid manual dataset descriptions. The release of the dataset and framework under an open license is a concrete asset for future data science agent research. The paper also reports cost ratios across approaches, which is practically relevant. However, the headline accuracy comparisons and temperature-sensitivity conclusions rest entirely on an unvalidated VLM judge, and the paper reports no statistical significance testing; these issues currently limit the strength of the conclusions.","major_comments":[{"comment":"The VLM-as-a-judge is insufficiently validated for the load-bearing accuracy claims. The paper states that only three of the 165 attempts were manually checked; since each of the 303 samples is evaluated over 165 attempts, this amounts to at most three of 49,995 judge decisions, and no inter-annotator agreement is reported. A judge error rate of even a few percent, if correlated with model, approach, or temperature, could change the rankings in Table 7 and the 'no significant temperature pattern' conclusion in Section 8. The fact that Gemini-2.5-Flash serves as both a judged model and the judge raises a self-preference concern that is not addressed. I ask the authors to validate the judge on a stratified random sample of outputs covering all models, approaches, query types, and temperatures, report agreement against human labels, and ideally use a second independent judge or a rubric-based human evaluation for at least a representative subset.","section":"Section 7, Appendix A (Prompt 5)"},{"comment":"The central claim that Claude-4-Sonnet with multi-code 'clearly outperformed' the rest is not supported by any significance testing. Several adjacent configurations differ by only 2–5 percentage points (e.g., Claude Multi-Code Clean at 58.36% vs. Claude SmolAgent Clean at 55.24%; Gemini Multi-Code Clean at 49.06% vs. Gemini Multi-Code Raw at 44.16%). With 303 samples and 11 temperature values, paired per-sample comparisons or bootstrap confidence intervals are needed to determine whether these gaps are distinguishable from noise. The same applies to the statement that no significant temperature pattern exists; the paper currently reports only descriptive statistics and plots.","section":"Section 8, Table 7"},{"comment":"The SmolAgent step limit is inconsistent: Section 7.3 states a limit of 8 steps, while Appendix B states a step limit of 5 for each query during inference. This is a reproducibility issue that affects the validity of all SmolAgent results, since the number of allowed steps directly determines how much context the agent can gather. The authors should state the correct value and, if both values were used in different runs, explain which runs used which limit.","section":"Section 7.3 vs. Appendix B"},{"comment":"The judge prompt specifies that numerical values 'rounded .2f values should match' and that errors beyond the second decimal after rounding can be ignored, but the paper does not analyze how sensitive the results are to this tolerance. Since many queries have numeric answers, a loose or inconsistent rounding rule could systematically favor certain output formats. Please report the distribution of judge decisions by answer type and, if possible, assess robustness to the tolerance parameter.","section":"Appendix A, Prompt 5"}],"minor_comments":[{"comment":"The reference 'as seen in Figure 46' appears in the main-text discussion of temperature effects, but Figure 46 is placed in Appendix I and appears to be a general results plot; the intended cross-reference is likely Figure 4 or a dedicated temperature figure. Please correct the reference.","section":"Section 8"},{"comment":"The heading 'Variations in results with Model and approach' is repeated twice in the same section; please rename the second occurrence to reflect its content, which concerns task-domain differences.","section":"Section 8"},{"comment":"The column header 'A VG.ACC' should read 'AVG.ACC' and should be defined in the caption as the average score over all attempts and temperature values for queries using that dataset.","section":"Table 4"},{"comment":"There is a typo in the annotation guidelines: 'categorizied' should be 'categorized'.","section":"Appendix C"},{"comment":"The captions for Figures 31–45 are empty or incomplete ('Incorrect response cause distribution -'); please provide descriptive captions that identify the model, approach, and query type for each figure.","section":"Appendix H"},{"comment":"The footnote explaining that SmolAgent was initially tested with one randomly chosen temperature value is unclear: it is not specified which temperature was chosen, and the sentence 'no change was observed in results between the Raw and Clean queries' would benefit from reporting the actual values.","section":"Section 7, footnote 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a useful benchmark resource, not a breakthrough. The genuinely new part is the annotation pipeline: 303 queries drawn from how clients actually used their commercial data science agent, with a task-type distribution matched to observed usage, and a structured context engineering schema (nested JSON of column stats, sample rows, summaries) instead of free-form dataset descriptions. That's a reasonable improvement over QRData and DSBench, and the raw/clean query pairs are a nice touch for testing data leakage. I also give credit for releasing under Apache 2.0, though I couldn't find a link to the actual dataset in the manuscript.\n\nThat said, the central evaluation has a load-bearing weakness. Every accuracy number in Table 7 and the temperature findings come from Gemini-2.5-Flash as VLM-as-a-judge, using the prompt in Appendix A. Section 7 says only 3 of the 165 attempts were manually checked. Since each of 303 samples goes through 165 attempts, that's 3 out of roughly 50,000 judge decisions. No inter-annotator agreement is reported. The judge is the same model family as one of the evaluated models, Gemini-2.5-Flash, so self-preference is a real possibility. The headline gaps are small: Claude multi-code raw at 59.92% versus Claude SmolAgent at 55.24%, with several adjacent configs within 2-3 points. A judge error rate of even a few percent, correlated with model or approach, could flip rankings and would also undermine the 'no temperature pattern' conclusion. The step-limit inconsistency (8 in 7.3 vs 5 in Appendix B) is minor by comparison but should be fixed.\n\nOn the positive side, the benchmark design is thoughtful: the data files have deliberately tricky features like non-datetime date columns and frequency changes, and the error analysis (70% wrong final answer, formatting errors concentrated in o4-mini) is informative. The routing suggestion is plausible as a direction, not proven.\n\nWho is this for? Practitioners choosing among LLM/data-agent configurations and benchmark builders wanting a more usage-grounded task mix. It deserves a serious referee, but my recommendation is to return it for major revision: release the dataset and evaluation code, validate the VLM judge against a proper human sample with agreement stats, run significance tests (or at least confidence intervals), and align the SmolAgent config across sections. It's a legitimate extension of existing benchmarks, with real value if the evaluation is shored up.","headline":"Useful usage-grounded data-science benchmark with a thoughtful context-engineering schema, but the headline rankings rest on a barely validated VLM judge and the dataset isn't released yet, so treat the numbers as provisional.","tokens_in":14029,"tokens_out":1944,"would_cite":false,"duration_ms":21421,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces DSBC, a 303-question benchmark built from real user logs, and reports that Claude-4-Sonnet generating multi-step code from engineered context clearly outperforms the other models and approaches.","keywords":["data science agents","benchmark","context engineering","large language models","LLM evaluation","temperature sensitivity","data leakage","VLM-as-a-judge"],"falsifier":"Grade a random sample of 30-50 responses across all fifteen model-approach-query-type setups with two independent human annotators against the ground-truth code and answer, and compare their verdicts with the VLM judge's Yes/No labels. If human-judge agreement falls below roughly 95%, or if the human-graded ranking does not place Claude-4-Sonnet multi-code-cell first, the paper's central performance claims are not established.","tokens_in":13058,"feed_emoji":"📊","tokens_out":15468,"duration_ms":137014,"temperature":0.7,"pith_summary":"This paper tries to establish that data-science agents should be judged on tasks real users actually ask, and it builds DSBC, a 303-question benchmark derived from logged usage of a commercial data-science agent. Every query is paired with a 'clean' rewrite that strips data leakage, and instead of hand-written dataset descriptions the model receives a standardized structured context listing column names, types, null counts, summaries, and the first five rows. Testing Claude-4-Sonnet, Gemini-2.5-Flash, and OpenAI-o4-Mini under three execution modes and eleven temperature settings, the paper reports that Claude-4-Sonnet with multi-step code generation clearly outperforms the field, that multi-step modes beat single-shot code generation, and that accuracy falls sharply as a query spans more task categories. If these results hold, model and approach choice matters far more than temperature when deploying a data-science agent.","feed_headline":"Claude-4-Sonnet wins real-user data-science benchmark","feed_subtitle":"303 queries drawn from live user logs: multi-step coding beats single-shot and agent frameworks.","key_machinery":"The load-bearing mechanism is context engineering: a standardized nested-JSON description of each data file — row and column counts, column names, data types, null counts, numeric min/max/mean/median/percentiles, categorical unique counts, datetime ranges and frequency, and the first five rows — that is inserted into the prompt in place of raw files or hand-written descriptions. Its job is to give the model enough information to solve the query while keeping sensitive data out of the context and removing the randomness that manually written descriptions introduce. A second mechanism is the raw/clean query pairing: each question exists in a leaked 'raw' form and a leakage-free 'clean' rewrite, letting the benchmark measure how much a prompt's hidden assumptions inflate apparent performance. The accuracy numbers themselves are produced by a VLM-as-a-judge setup in which Gemini-2.5-Flash returns a single-word Yes/No verdict in a JSON schema.","core_discovery":"On the paper's own terms, the central discovery is a stable set of performance disparities on realistic queries: Claude-4-Sonnet using a multi-code-cell approach (two to three code snippets generated in one pass, then executed in sequence) reaches the highest overall accuracy of any setup (roughly 58-60% averaged across temperatures, peaking just above 62%), ahead of the same model steering the SmolAgent agent framework (about 55%) and far ahead of single-cell direct generation, where OpenAI-o4-Mini is the best of the three (about 31%). Multi-step code generation and SmolAgent both clearly beat single-cell generation, temperature shows no systematic effect for any model-approach combination, and Gemini-2.5-Flash is the most temperature-sensitive model. The paper also reports that accuracy drops from 55.9% on queries covering one task category to 43.1% on two-category queries and 25.5% on three-category queries, and that roughly 70% of failures are incorrect final responses rather than code, data-access, or formatting errors.","pith_inferences":["The rankings and temperature findings rest on a judge that was manually validated on 3 of 165 attempts with no reported inter-annotator agreement; an obvious next step is to re-grade a larger random sample with independent human annotators and re-rank the setups on human verdicts.","The benchmark's design makes a comparison the authors did not run: holding queries fixed and swapping the structured JSON context for a hand-written description would isolate how much of the measured performance is attributable to the context format itself.","The accuracy staircase across one, two, and three task categories hints at compounding compositional failure — errors in one sub-task cascade into later ones — which a progress measure could track by scoring sub-task plans separately from final answers."],"forward_implications":["A standardized structured context can replace both raw data files and hand-written descriptions in benchmarks and deployments, addressing privacy concerns at the cost of a fixed metadata format.","The multi-code-cell advantage over SmolAgent — at 1.8x versus 3.2x the single-cell cost — implies that structured multi-step generation can match or beat a full agent framework for less money.","Because accuracy falls from 55.9% on one-category queries to 25.5% on three-category queries, multi-skill queries are the bottleneck that future agents must target, not single-skill ones.","The raw-versus-clean gap shows that prompt-level data leakage can distort measured performance, so benchmarks should ship leakage-free rewrites alongside original queries.","Task-category and model-approach variance suggest a query-routing policy — matching each incoming query to the cheapest model and approach that handles its category well — could preserve most accuracy while cutting inference cost."],"supporting_citations":[{"why":"Supplies Claude-4-Sonnet, one of the three evaluated models and the setup the paper reports as the top performer.","marker":"(Anthropic, 2025)"},{"why":"Gemini-2.5-Flash is both an evaluated model and the VLM judge that scores every response.","marker":"(Team, 2025)"},{"why":"Supplies OpenAI-o4-Mini, the third evaluated model, which leads the single-code-cell setting.","marker":"(OpenAI, 2025)"},{"why":"Supplies SmolAgent, the agent framework used as one of the three execution approaches.","marker":"(Roucher et al., 2025)"},{"why":"ReAct is the reasoning-and-acting loop on which the single- and multi-step code generation approaches build.","marker":"(Yao et al., 2023)"},{"why":"The context engineering survey motivates replacing hand-written dataset descriptions with a standardized structured context.","marker":"(Mei et al., 2025)"},{"why":"DS-1000 is the early data-science code benchmark used as a contrast in the positioning table for DSBC.","marker":"(Lai et al., 2023)"},{"why":"DSBench supplies the Excel-workflow benchmark comparison that frames the new benchmark's design choices.","marker":"(Jing et al., 2025)"}],"fun_headline_variants":["Claude-4-Sonnet multi-step beats SmolAgent on real queries","Multi-step code tops agent frameworks in data-science benchmark","Real-user data benchmark: multi-step coding wins over agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every accuracy number and ranking in the paper comes from a vision-language model judge (Gemini-2.5-Flash) that was checked against human judgment on only 3 of the 165 attempts, with no inter-annotator agreement reported, so even a small rate of judge error could change the rankings and the temperature findings.","fun_headline_variants_meta":{"raw":{"variants":["Claude-4-Sonnet multi-step beats SmolAgent on real queries","Multi-step code tops agent frameworks in data-science benchmark","Real-user data benchmark: multi-step coding wins over agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3515,"prompt_tokens":961,"completion_tokens":2554,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2496}},"tokens_in":577,"tokens_out":2554,"duration_ms":22688,"temperature":1.0,"reasoning_tokens":2496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:48:33.905989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Grade a random sample of 30-50 responses across all fifteen model-approach-query-type setups with two independent human annotators against the ground-truth code and answer, and compare their verdicts with the VLM judge's Yes/No labels. If human-judge agreement falls below roughly 95%, or if the human-graded ranking does not place Claude-4-Sonnet multi-code-cell first, the paper's central performance claims are not established.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Claude-4-Sonnet, one of the three evaluated models and the setup the paper reports as the top performer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies OpenAI-o4-Mini, the third evaluated model, which leads the single-code-cell setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SmolAgent, the agent framework used as one of the three execution approaches."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DSBench supplies the Excel-workflow benchmark comparison that frames the new benchmark's design choices."}],"review_version":1}