{"id":"abcfaf35-744c-4a8f-b0d8-72f6850e0441","arxiv_id":"2505.13291","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper presents TimeSeriesGym, a benchmark of 34 time series ML engineering challenges for AI agents, and shows that current agents produce valid solutions in 57.3% of tasks but reasonable ones in only 12.5%.","lead":"TimeSeriesGym is a new open-source benchmark that tests AI agents on 34 time series machine learning engineering tasks, from Kaggle-style forecasting to code migration. It scores agents on submissions, code, and models using both numerical metrics and LLM judges, and includes tools for generating new challenges.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All headline rates (12.5–38.9% reasonable, time/reminder non-effects) rest on a 'reasonable submission' label assigned by unvalidated manual/LLM log inspection; without inter-rater agreement or threshold sensitivity, the benchmark's key quantitative claims are not yet supported.","rationale":"The reader's weakest assumption identifies the unvalidated 'reasonable submission' label, and I agree that is the load-bearing point. My stress-test adds two sharpenings: (1) for Kaggle challenges the threshold itself is suspect because TimeSeriesGym uses different splits and reimplemented graders, so 'above original leaderboard median' is not a calibrated quantile of the benchmark's own metric; (2) for non-Kaggle challenges, manual/LLM log inspection is a binary judgment whose noise is amplified by the small Lite n. A single flip moves rates by ~17 points, enough to invert several Table 2 comparisons (e.g., 12.5% vs 27.8% for AIDE with/without Lite selection, or time-scaling rows). The paper's own Section 5 acknowledges 'inherent limitations' of the evaluation approach, and Appendix E rates judge grading as only 'moderate' reproducibility, but no reliability evidence is supplied. I do not think this invalidates the benchmark's construction or its usefulness as a resource; it does mean the headline model-comparison claims are conditional on a validation that is absent. Hence the reader's CONDITIONAL verdict should stand, with the revision requirement being a reliability analysis of the reasonable-submission metric.","tokens_in":17892,"tokens_out":6565,"duration_ms":60638,"concrete_test":"Audit every 'reasonable' label behind Tables 2 and 9. For non-Kaggle challenges, take the raw agent logs for all TimeSeriesGym-Lite runs (6 challenges x 3 seeds x each configuration) and have two independent human raters, blind to scaffold/model, apply the paper's definition; also run the LLM judge on the same logs. Report Cohen's kappa and LLM-human agreement, then recompute the Table 2 reasonable rates using majority human labels; if any rate shifts by one challenge (~17 points) or kappa < 0.6, the headline rates and non-effect conclusions should be presented as conditional on the labeling procedure. For Kaggle challenges, recompute the 'above median' threshold on TimeSeriesGym's own test splits instead of the original leaderboard and check whether the 12.5% full-benchmark reasonable rate survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative findings all pass through the 'reasonable submission' label defined in Section 4 (Metrics). For Kaggle challenges the label means scoring above the original competition's public-leaderboard median; however, TimeSeriesGym resplits data and reimplements grading (Section 5, 'Defining and measuring success'), so the original median is not obviously the 50th percentile of the benchmark's own score distribution. For non-Kaggle challenges, 'reasonable' is determined by examining agent logs manually or with an LLM-as-a-judge. The paper reports no inter-rater agreement, no LLM-vs-human concordance, and no threshold sensitivity. This matters because the headline observations in Table 2 are the entire empirical payload: AIDE > OpenHands, o3's high valid rate, Claude's 38.9% reasonable rate, and the null effects of more time and reminders. TimeSeriesGym-Lite has only six challenges, so one re-labeled challenge changes a reasonable rate by about 17 percentage points. The authors even concede the bar is low and that current evaluation approaches have 'inherent limitations,' but they do not quantify how much label noise could move the reported rates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TimeSeriesGym, an open-source benchmarking framework for evaluating AI agents on time series machine-learning engineering tasks. The framework combines Kaggle-derived challenges, original challenges built on research repositories, and derived variants, and supports evaluation of multiple artifact types (prediction files, code, trained models) using both quantitative metrics and LLM-based qualitative judgment. The authors report experiments on a six-challenge Lite subset with several scaffolds (AIDE, OpenHands) and models (GPT-4.1, o3, Claude 3.7), observing that AIDE outperforms OpenHands, that o3 yields more valid submissions, that Claude 3.7 produces the most 'reasonable' submissions, and that additional time or step-wise reminders do not consistently improve performance. They also provide a familiarity analysis intended to address contamination concerns and describe tools for generating new challenges at scale.","tokens_in":18281,"tokens_out":3522,"duration_ms":32605,"significance":"If the evaluation methodology is sound, TimeSeriesGym would be a useful community resource: it is open-sourced under a permissive license, ships machine-readable challenge descriptions and grading code, supports multiple agent scaffolds, and explicitly targets multimodal artifacts and holistic evaluation, which several existing ML-agent benchmarks do not. The paper's main quantitative claims, however, rest on a 'reasonable submission' label that is assigned by manual or LLM inspection without reported reliability checks, and the challenge count is inconsistent across the abstract, Table 1, and Table 3. Because the benchmark artifact itself is concrete and reproducible, these issues are fixable, but they currently preclude accepting the headline empirical findings as stated.","major_comments":[{"comment":"The headline rates (12.5–38.9% reasonable submissions, and the claimed non-effects of time and reminders) all depend on the 'reasonable submission' label, but the label is not validated. For Kaggle challenges the threshold is the original competition's public-leaderboard median; as Section 5 ('Defining and measuring success') concedes, TimeSeriesGym resplits data and reimplements grading, so that median is not necessarily the 50th percentile of the benchmark's own score distribution. For non-Kaggle challenges the label comes from manual or LLM-as-a-judge inspection of logs, with no inter-rater agreement, no LLM-vs-human concordance, and no threshold sensitivity analysis. Since TimeSeriesGym-Lite has only six challenges, one re-labeled challenge changes a reasonable rate by about 17 percentage points, so the reported ordering of models and the null effects of more time are not robustly established. The paper should report inter-annotator agreement, a calibration study of the LLM judge, and a sensitivity analysis showing how the rates in Table 2 move under plausible labeling noise.","section":"Section 4 (Metrics) and Table 2"},{"comment":"The number of challenges is reported inconsistently. The abstract and Section 3 state 34 challenges, but Table 3 lists 12 Kaggle + 14 Originals + 6 Derived = 32 challenges. Table 4 (sources/licenses) omits two of the Originals listed in Table 3 (the two CSDI challenges), and Table 1 reports '23+ 23+' for TimeSeriesGym, which appears to conflate data sources with challenge counts. These discrepancies must be reconciled, and the final counts in the abstract, Section 3, and all tables should agree exactly.","section":"Section 3 and Table 3"},{"comment":"The paper states that o3 achieves 'significantly higher valid submission rates (94.4%)' and that Claude 'produces the most reasonable submissions,' but no statistical tests are reported for these comparisons, which are based on only three seeds. Given the small sample and the label-noise issue noted above, phrases such as 'significantly' should be backed by either appropriate tests or explicit acknowledgment that the differences are not statistically tested.","section":"Section 4.1 (Observations)"}],"minor_comments":[{"comment":"The claim that the benchmark is 'scalable' is supported only by an anecdote that 'several new challenges' were created in two hours; please quantify this claim (how many challenges, by whom, with what quality) or soften it.","section":"Section 3.1 (Design Choices)"},{"comment":"The KS test p-value of 0.363 is reported for the difference between familiarity distributions, but the sample sizes (TimeSeriesGym vs. MLE-bench) and the power of the test are not discussed; a null result with small samples should be interpreted cautiously.","section":"Section 4 (Familiarity analysis, Fig. 2)"},{"comment":"The judge-based grading approach (G-Eval) is described, but there is no evidence of its agreement with human experts; a short validation study would strengthen the claim that LLM-as-a-judge is a reliable evaluation component.","section":"Appendix E"},{"comment":"Several cells contain 'N/A' without explaining whether the agent produced no valid submission or the configuration was not run; please distinguish these cases.","section":"Table 9"},{"comment":"There are minor typographical issues, e.g., 'Averge Cost' in Table 6 and 'Mean Squarred Error' in Table 3, and the caption of Table 5 says 'six diverse time series challenges' while the table lists six, which is consistent; please proofread.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The self-citation aspect is worth watching: several original challenges are built on the authors' own MOMENT and TimeSeriesExam code, which gives those repositories a privileged role in the benchmark. This is not disqualifying, but it should be explicitly disclosed in the paper. The count inconsistencies and the lack of validation for the 'reasonable' label are the main blockers; if the authors can address them with additional analysis and a revised manuscript, the paper could become accept-worthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nIf you work on ML engineering agents, you should know TimeSeriesGym exists. It's an open-source, time-series-focused benchmark with 34 challenges spanning forecasting, classification, anomaly detection, and code migration, plus a lite subset that costs about $8 per agent run. The tooling for generating new challenges is a real plus; that's how you keep a benchmark alive.\n\nWhat's new: it's the first time-series-specific agent benchmark, and it evaluates multiple artifacts—prediction files, code, models—using both quantitative metrics and LLM-as-a-judge. The design is thoughtful: agent-agnostic scaffolds, per-challenge graders, and an honest discussion of contamination, plagiarism, and the limitations of current evaluation. The experimental setup is reasonable for a first pass: three seeds, two scaffolds, three models, and cost reporting.\n\nThe soft spots are real but not fatal. The headline rates ('reasonable submissions') depend on a label that isn't well validated. For Kaggle challenges, 'above median on the public leaderboard' is used, but the authors resplit the data and reimplement grading, so the original median may not correspond to the 50th percentile of their own score distribution. For non-Kaggle challenges, 'reasonable' is determined by manual or LLM inspection of logs, with no inter-rater agreement and no threshold sensitivity analysis. Since the Lite set has only six challenges, one relabeled challenge shifts a rate by about 17 percentage points. The authors acknowledge the bar is low, but they don't quantify how much label noise could move the results. This matters because the most interesting findings—that more time doesn't help and reminders sometimes hurt—all pass through that label.\n\nThere's also a minor inconsistency: the abstract says 34 challenges; Table 3 lists 32. And the 'first open source benchmark' claim is a bit strong given MLE-bench and MLAgentBench are open-source, though the time-series focus is genuinely new.\n\nNone of this undermines the core artifact. The benchmark itself is concrete, reproducible, and useful, and the central claim holds up. The paper deserves a serious referee, not a desk reject. In revision I'd want inter-rater agreement on the reasonable label, a sensitivity analysis with stricter criteria, and a reconciliation of the challenge count. I'd bring it to a reading group and cite it when writing about agent evaluation.\n\nBest","headline":"A useful open-source benchmark for time-series ML agents; the 'reasonable submission' label needs validation before its headline rates should be trusted.","tokens_in":18670,"tokens_out":3483,"would_cite":true,"duration_ms":29781,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TimeSeriesGym gives AI agents a realistic test: on 34 time-series ML engineering challenges, even frontier models manage reasonable submissions only 12.5%–38.9% of the time.","keywords":["AI agents","benchmarking","time series","machine learning engineering","LLM-as-a-judge","multimodal evaluation","scalable benchmark","agentic scaffolds"],"falsifier":"Take a random sample of TimeSeriesGym-Lite runs, have two human raters independently classify each submission as reasonable or not using the paper's stated criteria, and compare their labels with each other and with the LLM judge's labels; if inter-rater agreement is low (for example, Cohen's kappa below 0.5), the reported 12.5%–38.9% reasonable-submission rates would be too unstable to support the paper's conclusions.","tokens_in":17749,"feed_emoji":"🕒","tokens_out":5694,"duration_ms":52075,"temperature":0.7,"pith_summary":"TimeSeriesGym is an open, agent-agnostic benchmark for evaluating AI agents on time-series machine learning engineering tasks. It aims to fix what the authors see as the narrowness of existing benchmarks, which focus on model building in well-defined settings and score only prediction files. The framework contains 34 challenges across 8 time-series problem types and more than 15 domains, drawing on Kaggle-style competitions, original tasks from research repositories, and derived variants that isolate specific skills such as data handling, hyperparameter tuning, code migration, and working with research code. The paper also introduces a two-faceted grading system that scores submission files, code, and models with both precise numeric metrics and LLM-as-a-judge assessment. Using this framework, the authors show that current state-of-the-art agents (AIDE and OpenHands scaffolding with GPT-4.1, o3, and Claude 3.7) produce valid outputs for most challenges but reasonable ones for only a minority, and that giving agents more time does not reliably improve performance.","feed_headline":"AI agents pass under 40% of time-series ML engineering tests","feed_subtitle":"TimeSeriesGym tests 34 real-world tasks; even top models earn 'reasonable' marks only 12.5–38.9% of the time.","key_machinery":"The central mechanism is the TimeSeriesGym challenge structure itself: each challenge bundles resources (datasets, code repositories, papers), a description file with instructions and hints, and a challenge-specific grading function. On top of this, the framework provides tools for generating new challenges at scale (for example, simulating missing data or creating derived variants) and a dual grading approach that combines exact, deterministic checks (regex, AST parsing, test cases) with judge-based evaluation using LLM chain-of-thought scoring following the G-Eval approach. The combination of diverse task sources (Kaggle competitions, research repositories, hand-crafted originals) with multimodal artifact grading and skill-isolating challenge design is what carries the benchmark's claim to be a comprehensive, scalable, and diagnostic testbed rather than another accuracy-only leaderboard.","core_discovery":"The paper claims that TimeSeriesGym provides a scalable, agent-agnostic benchmarking environment that evaluates multimodal agent outputs (prediction files, code, models) across specific ML-engineering skills, combining quantitative metrics with LLM-based qualitative judgment. Its main empirical finding is that frontier agents still fail at these tasks: with AIDE and GPT-4.1 on the full benchmark, valid submissions were produced for 57.3% of challenges but reasonable submissions for only 12.5% of them. On the cheaper TimeSeriesGym-Lite subset, the reasoning model o3 achieved 94.4% valid submissions, while Claude 3.7 led in reasonable submissions at 38.9%. The paper also observes that doubling or tripling the time and step budget did not consistently improve results, and that removing the step-wise time reminders sometimes increased the rate of reasonable submissions, suggesting agents do not use extra time strategically. These findings are offered as evidence that current agents understand and solve realistic time-series tasks far less well than their raw valid-submission rates suggest.","pith_inferences":["The headline capability rates rest on the subjective judgment of what counts as a 'reasonable' submission; since the paper does not report inter-rater agreement between human and LLM judges, a reliability audit (double-coding a sample of logs) is a natural next step that could shift those numbers.","The skill-isolating design could be extended to produce diagnostic profiles for individual agents—for example, radar charts of which skills each agent lacks—which the authors gesture toward but do not implement.","Since the authors state the framework is modality-agnostic in principle, the same challenge-generation and grading tools could be adapted to image, text, or audio ML engineering tasks, though that extension is not demonstrated in this paper."],"forward_implications":["TimeSeriesGym can serve as a reusable testbed for tracking progress in ML engineering agents; its lightweight subset, TimeSeriesGym-Lite, makes rapid and inexpensive evaluation possible at roughly $8 per run.","Because the framework supports generating new challenges and skill-specific variations, it can evolve to mitigate pretraining contamination even after the current version becomes public.","The multimodal, skill-based evaluation provides actionable feedback beyond a single score, helping practitioners identify specific weaknesses such as poor data handling or inability to use research code.","The observed failure modes of AIDE (single-file emphasis, missed README documentation) and OpenHands (greedy, linear file scanning) indicate that better scaffolds—ones that explore alternative solutions and consult documentation—are needed for repository-level ML tasks.","Collected agent trajectories can be used as training data for post-training refinement, enabling the benchmark to act as a data flywheel for improving agents."],"supporting_citations":[{"why":"MLE-bench provides the methodological template the paper follows, including agent-agnostic challenge design, scaffold modifications, and the familiarity/contamination analysis.","marker":"[3]"},{"why":"AIDE is the primary agent scaffold used in the experiments, supplying the code-exploration mechanism whose failure modes the paper analyzes.","marker":"[11]"},{"why":"OpenHands is the second scaffold evaluated, providing the comparison that leads to the finding that AIDE performs better on most TimeSeriesGym challenges.","marker":"[21]"},{"why":"SUPER motivates the holistic evaluation approach that combines outcome-based scoring with qualitative code inspection, which TimeSeriesGym extends to time-series tasks.","marker":"[1]"},{"why":"TimeSeriesExam inspires the scalability and comparative-characterization ideas, particularly the use of structured task generation and the envisioned automated pairwise agent comparisons.","marker":"[2]"},{"why":"G-Eval supplies the LLM-as-a-judge framework used in the judge-based grading component of the two-faceted evaluation methodology.","marker":"[14]"}],"fun_headline_variants":["TimeSeriesGym: AI agents fail real-world time-series ML tasks","Only 12.5% of AI agent outputs judged reasonable on TimeSeriesGym","More time, same struggle: AI agents on time-series engineering","TimeSeriesGym benchmark shows LLM agents miss most engineering tasks","AI agents ace 'valid' but flunk 'reasonable' on time-series tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline results depend on treating 'reasonable submission' as a reliably recognizable category: someone (a human reader or an LLM judge) has to read an agent's logs and decide whether the attempt was a genuine solution, and the paper does not report how consistent those judgments are.","fun_headline_variants_meta":{"raw":{"variants":["TimeSeriesGym: AI agents fail real-world time-series ML tasks","Only 12.5% of AI agent outputs judged reasonable on TimeSeriesGym","More time, same struggle: AI agents on time-series engineering","TimeSeriesGym benchmark shows LLM agents miss most engineering tasks","AI agents ace 'valid' but flunk 'reasonable' on time-series tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000314,"raw_usage":{"total_tokens":1808,"prompt_tokens":997,"completion_tokens":811,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":613,"tokens_out":811,"duration_ms":7452,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:15:11.671778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of TimeSeriesGym-Lite runs, have two human raters independently classify each submission as reasonable or not using the paper's stated criteria, and compare their labels with each other and with the LLM judge's labels; if inter-rater agreement is low (for example, Cohen's kappa below 0.5), the reported 12.5%–38.9% reasonable-submission rates would be too unstable to support the paper's conclusions.","supporting_citations":[{"cited_title":"SUPER: Evaluating agents on setting up and executing tasks from research repositories","cited_arxiv_id":null,"evidence_quote":"SUPER motivates the holistic evaluation approach that combines outcome-based scoring with qualitative code inspection, which TimeSeriesGym extends to time-series tasks."},{"cited_title":"TimeSeriesExam: A time series understanding exam","cited_arxiv_id":null,"evidence_quote":"TimeSeriesExam inspires the scalability and comparative-characterization ideas, particularly the use of structured task generation and the envisioned automated pairwise agent comparisons."},{"cited_title":"G-eval: Nlg evaluation using gpt-4 with better human alignment","cited_arxiv_id":null,"evidence_quote":"G-Eval supplies the LLM-as-a-judge framework used in the judge-based grading component of the two-faceted evaluation methodology."}],"review_version":1}