Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

TAM-Bench claims to deliver a diverse, realistic, and structured benchmark for LLM-based agents, with automated task acquisition, objective difficulty modeling, and multi-dimensional evaluation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A benchmark of 150 AutoML competition tasks built by a web-agent pipeline, with leaderboard-derived difficulty labels and multi-metric evaluation.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful 150-task benchmark and difficulty model, but the rank-based evaluation rests on an unvalidated score mapping and the task list contradicts the stated contamination filter. the 4 major comments →

arxiv 2509.09321 v1 pith:TQVPIUIU submitted 2025-09-11 cs.AI

Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization

Hangyi Jia , Yuxi Qian , Hanwen Tong , Xinhui Wu , Lin Chen , Feng Wei This is my paper
classification cs.AI
keywords AutoML benchmarkLLM agentstask acquisitionleaderboard difficultymulti-dimensional evaluationweb automationmodality coveragereward hacking
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces TAM-Bench, a benchmark for evaluating large language model (LLM) agents on end-to-end machine learning tasks. Its central claim is that such a benchmark can be built largely automatically and assessed more objectively than existing ones. Three mechanisms carry the argument: a browser-driving web agent that scrapes and schema-unifies competition tasks from platforms like Kaggle; a difficulty model derived solely from leaderboard signals (normalized mean score, best score, and participant count); and a multi-metric evaluation that tracks rank percentile, submission format validity, and LLM-judged constraint compliance. The authors construct 150 curated tasks across six data modalities, release Lite, Medium, and Full subsets of sizes 18, 54, and 150, and run experiments on the Lite subset with two agent frameworks and two base LLMs. The paper's contribution is a claim that benchmark construction for AutoML agents can be scaled, made more objective, and made more resistant to reward hacking.

Core claim

The paper's central claim is that a benchmark for AutoML agents can be constructed largely automatically and evaluated in a way that is more objective and comprehensive than existing benchmarks. The authors design a browser-automation pipeline that converts competition pages into a unified schema, a difficulty score based on the normalized mean and best scores plus the log participant count on the private leaderboard, and an evaluation framework that combines the agent's rank percentile, the validity of its submission file, and an LLM-judged constraint pass rate. They report that this pipeline produced 150 tasks across six data modalities, with a balanced Lite subset of 18 tasks, and that in

What carries the argument

The central object is the three-tier benchmark itself, built on three mechanisms: (1) a web-agent acquisition system using a Browser-Use style controller layered over Playwright to scrape and schema-unify task descriptions; (2) a leaderboard-derived difficulty score computed from normalized mean score, normalized best score, and log participant count, which maps continuous scores into Easy/Medium/Hard bins; and (3) a multi-metric evaluation pipeline that converts a submission's raw score into a leaderboard RankPct, then audits the submission file for format validity and uses an LLM judge to check constraint adherence. The RankPct is the load-bearing metric: all difficulty labels and performa

Load-bearing premise

The load-bearing premise is that the locally reconstructed test sets produce scores that are directly comparable to the original competition leaderboard scores, so that an agent's local score can be converted into a rank percentile and difficulty labels built from leaderboards remain valid.

What would settle it

A concrete test: for a set of tasks covering all six modalities, take a fixed set of predictions (e.g., the competition's sample submission and several leaderboard submissions), run them through both the official evaluation system and the TAM-Bench local pipeline, and compare the resulting scores. If for any task the two scores diverge enough to change the percentile rank by a meaningful margin, or if the scores are not monotonically related, the RankPct and difficulty labels built from leaderboards would not be trustworthy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If TAM-Bench is right, benchmark construction for ML agents can be scaled by scraping rather than manual curation, lowering cost and extending coverage to new domains.
  • The leaderboard-based difficulty model provides an objective, extensible alternative to expert time estimates, making difficulty labels reusable across platforms.
  • The multidimensional evaluation framework (rank, format, constraints) would reduce reward hacking, since an agent that maximizes score while ignoring format or business constraints is penalized.
  • The modality and difficulty stratified subsets (Lite/Medium/Full) allow evaluations to be tuned to available compute and provide fine-grained diagnosis of an agent's strengths and weaknesses.
  • The experiments suggest that reliability (format compliance, submission validity) is a separate axis from raw performance, and that these axes can diverge between base models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The split-reconstruction and leaderboard-placement procedure could be applied to any historically hosted competition with a public training set and a private leaderboard; the consistency check the paper uses (comparing sample submission scores on official vs local evaluation) is a generalizable validation method, and if applied more broadly would test whether the comparability assumption holds acr
  • The difficulty score's dependence on participant count couples task difficulty with community engagement, which may conflate 'hard for novices' with 'popular' – a potential confound for tasks with few participants.
  • The LLM-as-a-judge constraint check could be extended to automatically generate adversarial constraints to stress-test agent compliance, since the current setup only checks the constraints that appear in the competition description.
  • If the benchmark is maintained and expanded, the automated pipeline could track the evolution of agent capabilities over time, serving as a living benchmark rather than a static snapshot.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces TAM-Bench, a 150-task benchmark for evaluating LLM-based agents on end-to-end ML tasks. The benchmark is built via a browser-automation and LLM pipeline that scrapes and structures competitions from Kaggle, AIcrowd, and Biendata. The authors propose a leaderboard-signal-based difficulty model (Eqs. 1--4) and a multi-dimensional evaluation framework (rank percentile, format compliance, constraint pass rate). They evaluate AIDE and OpenHands with GPT-4.1 and DeepSeek-V3 on the 18-task Lite subset and report per-task rank percentages, difficulty-stratified averages, and modality-stratified averages.

Significance. If validated, the automated acquisition pipeline and the 150-task coverage would be a useful extension of existing benchmarks such as MLEBench, and the leaderboard-driven difficulty calibration is a genuinely scalable alternative to expert annotation. The multi-metric evaluation framework is a sensible step toward preventing reward hacking. The paper also provides detailed per-task result tables and a public repository, which are commendable. However, the central performance metric -- RankPct -- depends on an unvalidated transfer of locally computed scores onto official leaderboards, and there are internal inconsistencies in the reported results. These issues must be resolved before the benchmark's headline claims can be accepted.

major comments (4)
  1. [Dataset Splitting and Evaluation Consistency / Evaluation Metrics] RankPct is computed by scoring an agent's submission on a reconstructed split of the original training set, then placing that score on the official competition leaderboard, whose ranks are based on the official private test set. Because official participants trained on the full original training set -- which contains TAM-Bench's local test subset -- the agent's local-test score and the participants' private-test scores are not directly comparable. The consistency check using sample submission.csv files is insufficient: sample submissions are often constant/placeholder predictions, so near-identical scores at a single point do not establish that local and official scores have the same scale or a monotonic relationship across the relevant range. This affects every RankPct value in Tables 2--4 and 6--9 and the Average Rank headline in Table 2. The paper needs either direct validation on mul
  2. [Case Highlight vs. Table 9] The case highlight claims that 'OpenHands + DeepSeek-V3' achieved first place on stanford-covid-vaccine with an MCRMSE of 0.30, surpassing the best human score of 0.34198. However, Table 9, which reports OpenHands with DeepSeek-V3, shows a dash for stanford-covid-vaccine, indicating that no valid submission was generated. Table 8, for AIDE with DeepSeek-V3, is the one that lists a raw score of 0.309915 and a rank percentage of 0.0006 for this task. This internal inconsistency makes the highlighted result unauditable and must be corrected.
  3. [Automated benchmark construction / filtering] The pipeline states that tasks are filtered to 'exclude those before 2023 to avoid contamination from pretraining corpora.' Yet Table 10 (Lite) and Table 12 (Full) include many tasks from well before 2023, e.g., mlsp-2013-birds, new-york-city-taxi-fare-prediction, denoising-dirty-documents, statoil-iceberg-classifier-challenge, and the-icml-2013-whale-challenge. Either the filter was not applied as described, the description is inaccurate, or the temporal cut-off refers to something else. Since contamination avoidance is a stated rationale for task selection, this contradiction needs to be resolved.
  4. [Equations (3)--(4) and difficulty thresholds] The difficulty score in Eqs. (3)--(4) uses hand-set weights (w1=0.4, w2=0.1, w3=0.5) and hand-set thresholds (0.6 and 0.85) with no sensitivity analysis or fitting procedure. The paper describes the method as 'objective' and 'automated,' which overstates the case. Because these labels determine the composition of the Lite/Medium/Full subsets and the difficulty-stratified analysis in Table 3, the stability of the taxonomy under plausible weight variations should be reported. The MLEBench confusion matrix provides useful external grounding but does not eliminate the arbitrariness of the chosen constants.
minor comments (5)
  1. [Throughout] The name is inconsistent: 'TAM Bench' in the abstract vs. 'TAM-Bench' elsewhere, and 'MLEBench' vs. 'MLE-Bench'. Missing spaces before citations also appear, e.g., 'AIDE(Jiang et al. 2025)'.
  2. [Table 2] The caption of Table 2 lists 'Average Rank (%)' but does not state that lower is better. The per-task tables state it; the summary table should too.
  3. [RankPct formula] RankPct is defined as rank_i / total_participants_i. If it is a percentage, it should be multiplied by 100; otherwise it should be called a rank fraction.
  4. [Appendix A.8] The text says 'Table 12 lists the 18 competitions included in the Full version,' but Table 12 lists 150 competitions. This appears to be a typo.
  5. [Table 11] The task 'the-icml-2013-whale-challenge-right-whale-redux' appears twice in the Medium version list, and the Full version in Table 12 also contains duplicates. Please deduplicate and double-check the task inventory.

Circularity Check

0 steps flagged

No significant circularity identified.

full rationale

The paper's central claims (automated task acquisition, leaderboard-based difficulty modeling, and multi-dimensional evaluation) do not reduce by construction to their own inputs. Difficulty scores (Eqs. 1-4) are hand-weighted functions of aggregate leaderboard statistics (NormMean, NormBest, participant count) and are validated against MLEBench's independent manual difficulty labels (Appendix A.2-A.4). The performance metric RankPct_i = rank_i / total participants_i is a rank transform of the agent's own score on the official leaderboard; the difficulty score never includes the agent's rank, and RankPct never enters the difficulty formula. Thus cross-tabulating RankPct by difficulty level is not a tautology. No parameter is fitted to a subset and then renamed as a prediction; the weights w1=0.4, w2=0.1, w3=0.5 and thresholds 0.6/0.85 are stated as empirical settings rather than fit to outcome labels. The paper contains no self-citations by the present authors, and no load-bearing uniqueness theorem is imported from prior work. The 'Dataset Splitting and Evaluation Consistency' check (sample submission scores nearly identical on official vs. reconstructed local pipelines) is a validity argument and may be too weak to establish full score-scale equivalence, but that is a correctness/robustness concern, not a circularity: the local test score is placed on the original leaderboard, not defined in terms of the agent's rank. Overall, the derivation chain is self-contained against external benchmarks and does not exhibit circular reasoning.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The benchmark construction is concrete and useful, but its central numerical claims rest on hand-set difficulty parameters, an untested comparability assumption for local splits, and LLM-generated annotations with no reported human verification. No physical or ontological entities are introduced.

free parameters (2)
  • Difficulty weights w1, w2, w3 = 0.4, 0.1, 0.5
    Hand-set in Eq. (3)-(4) as 'empirically set'; no sensitivity analysis or fitting procedure is described, and these weights determine all difficulty bins.
  • Difficulty thresholds = 0.6, 0.85
    Hand-chosen cutoffs in the difficulty partitioning step; the paper does not explain how they were selected, and they map continuous difficulty scores to Easy/Medium/Hard labels.
axioms (3)
  • domain assumption Reconstructed train/test splits are representative of the original private test sets.
    Used in 'Dataset Splitting and Evaluation Consistency' to justify computing local scores and mapping them to official leaderboard ranks; only a qualitative 'nearly identical' check is offered.
  • domain assumption Leaderboard participant counts and score dispersion are valid proxies for task difficulty.
    Foundation of the difficulty score in Eq. (3)-(4); validated only by agreement with MLEBench manual labels for shared tasks, not by independent difficulty measurements.
  • domain assumption GPT-4o schema extraction and LLM-as-judge constraint checks accurately reflect task requirements.
    Used to build schemas and compute Constraint Pass Rate; no human audit, inter-annotator agreement, or error analysis is reported.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization." pith.science (2026). https://pith.science/paper/TQVPIUIU

@misc{pith2026250909321,
  author       = {Pith},
  title        = {Pith review of: Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TQVPIUIU}},
  note         = {Machine review of arXiv:2509.09321}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent advances in large language models (LLMs) have enabled the emergence of general-purpose agents for automating end-to-end machine learning (ML) workflows, including data analysis, feature engineering, model training, and competition solving. However, existing benchmarks remain limited in task coverage, domain diversity, difficulty modeling, and evaluation rigor, failing to capture the full capabilities of such agents in realistic settings. We present TAM Bench, a diverse, realistic, and structured benchmark for evaluating LLM-based agents on end-to-end ML tasks. TAM Bench features three key innovations: (1) A browser automation and LLM-based task acquisition system that automatically collects and structures ML challenges from platforms such as Kaggle, AIcrowd, and Biendata, spanning multiple task types and data modalities (e.g., tabular, text, image, graph, audio); (2) A leaderboard-driven difficulty modeling mechanism that estimates task complexity using participant counts and score dispersion, enabling scalable and objective task calibration; (3) A multi-dimensional evaluation framework incorporating performance, format compliance, constraint adherence, and task generalization. Based on 150 curated AutoML tasks, we construct three benchmark subsets of different sizes -- Lite, Medium, and Full -- designed for varying evaluation scenarios. The Lite version, with 18 tasks and balanced coverage across modalities and difficulty levels, serves as a practical testbed for daily benchmarking and comparative studies.

Figures

Figures reproduced from arXiv: 2509.09321 by Feng Wei, Hangyi Jia, Hanwen Tong, Lin Chen, Xinhui Wu, YuXi Qian.

Figure 1
Figure 1. Figure 1: An overall illustration of TAM-Bench more balanced domain and modality representation. All can￾didate tasks are drawn from a larger pool of competitions and pass a strict filtering pipeline: (1) exclude those before 2023 to avoid contamination from pretraining corpora; (2) remove tasks without publicly available datasets; (3) discard tasks lacking ground-truth test labels or reproducible data splits; and (… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of Task Difficulty by Modality [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Difficulty Confusion Matrix: TAM-Bench vs. MLE-Bench [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of difficulty levels between MLE-Bench and TAM-Bench [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: This graph shows the 24 competitions in MLE-Bench that changed in difficulty, and the remaining 51 that [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: example of constraint pass [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery

    cs.CL 2026-02 reject novelty 6.0

    DBench-Bio builds a dynamic biology benchmark from post-release abstracts, but LLM-generated gold answers and unverified per-model temporal separation undermine its claim to measure knowledge discovery.

  2. ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision

    cs.AI 2026-08 conditional novelty 5.0

    Partial LLM-agent evaluations can support a promote/reject decision after 15–25% of tasks on some benchmarks, but not others; reports should state the decision rule and unresolved count.

  3. From Question Answering to Task Completion: A Survey on Agent System and Harness Design

    cs.AI 2026-06 unverdicted novelty 4.0

    Survey framing LLM agents as model-plus-harness systems, decomposing harness responsibilities, mapping them to tasks, and highlighting open challenges in evaluation, safety, and co-evolution.

Reference graph

Works this paper leans on

12 extracted references · 8 linked inside Pith · cited by 3 Pith papers

  1. [2]

    arXiv:2412.19437

    DeepSeek-V3 Technical Report. arXiv:2412.19437. Feng, D.; Dai, Y .; Huang, J.; Zhang, Y .; Xie, Q.; Han, W.; Chen, Z.; Lopez-Lira, A.; and Wang, H

    Pith/arXiv arXiv

  2. [3]

    Based on thebest solution.pyand the special instructionfield, we apply an LLM-as-judge method

    Must-use features: Consider extracting acoustic features such as MFCCs (Mel-Frequency Cepstral Coefficients) or spectrograms for model input." 26} Table 5: The structured schema of tensorflow-speech-recognition-challenge result task A.6 Example of constraint pass evaluation Figure 6 shows a sample case used to evaluate theConstraint Pass Rate. Based on th...

    2013

  3. [4]

    arXiv:2503.23278

    Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions. arXiv:2503.23278. Huang, Q.; V ora, J.; Liang, P.; and Leskovec, J

    Pith/arXiv arXiv

  4. [5]

    arXiv:2310.03302

    MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation. arXiv:2310.03302. Jiang, Z.; Schmidt, D.; Srikanth, D.; Xu, D.; Kaplan, I.; Ja- cenko, D.; and Wu, Y

    Pith/arXiv arXiv

  5. [6]

    arXiv:2502.13138

    AIDE: AI-Driven Exploration in the Space of Code. arXiv:2502.13138. Liu, Z.; Cai, Y .; Zhu, X.; Zheng, Y .; Chen, R.; Wen, Y .; Wang, Y .; E, W.; and Chen, S. 2025a. ML-Master: To- wards AI-for-AI via Integration of Exploration and Reason- ing. arXiv:2506.16499. Liu, Z.; Chai, J.; Zhu, X.; Tang, S.; Ye, R.; Zhang, B.; Bai, L.; and Chen, S. 2025b. ML-Agent...

    Pith/arXiv arXiv

  6. [7]

    https://openai

    GPT-4.1 Technical Report. https://openai. com/index/gpt-4-1/. Accessed: 2025-08-02. Ou, Y .; Luo, Y .; Zheng, J.; Wei, L.; Qiao, S.; Zhang, J.; Zheng, D.; Chen, H.; and Zhang, N

    2025

  7. [8]

    arXiv:2506.10974

    AutoMind: Adaptive Knowledgeable Agent for Automated Data Sci- ence. arXiv:2506.10974. Wang, X.; Li, B.; Song, Y .; Xu, F. F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y .; Li, B.; Singh, J.; Tran, H. H.; Li, F.; Ma, R.; Zheng, M.; Qian, B.; Shao, Y .; Muennighoff, N.; Zhang, Y .; Hui, B.; Lin, J.; Brennan, R.; Peng, H.; Ji, H.; and Neubig, G

    arXiv

  8. [9]

    arXiv:2407.16741

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arXiv:2407.16741. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y

    Pith/arXiv arXiv

  9. [12]

    A dash (“-”) indicates that nosubmission.csvwas generated or the generated file failed to meet the required format for evaluation

    with GPT-4.1 across modalities and difficulties, including raw scores and rank percentages (lower is better). A dash (“-”) indicates that nosubmission.csvwas generated or the generated file failed to meet the required format for evaluation. Task Name Modality Difficulty Raw Score Rank Percentage new-york-city-taxi-fare-prediction Tabular Easy - - Binary P...

    arXiv 2013

  10. [2023]

    task_type

    ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. A.1 Distribution of Task Difficulty by Modality Figure 2 presents the difficulty distribution across different data modalities for the 150 tasks in the full version of TAM-Bench. As shown, TAM-Bench covers all three difficulty levels—easy,medium, andhard—across six data modaliti...

    Pith/arXiv arXiv

  11. [2024]

    arXiv:2310.00566

    Empower- ing Many, Biasing a Few: Generalist Credit Scoring through Large Language Models. arXiv:2310.00566. Hou, X.; Zhao, Y .; Wang, S.; and Wang, H

    Pith/arXiv arXiv

  12. [2025]

    arXiv:2410.07095

    MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineer- ing. arXiv:2410.07095. DeepSeek-AI; Liu, A.; Feng, B.; Xue, B.; Wang, B.; Wu, B.; Lu, C.; Zhao, C.; Deng, C.; Zhang, C.; Ruan, C.; Dai, D.; Guo, D.; Yang, D.; Chen, D.; Ji, D.; Li, E.; Lin, F.; Dai, F.; Luo, F.; Hao, G.; Chen, G.; Li, G.; Zhang, H.; Bao, H.; Xu, H.; Wang, H.; Zhan...

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.