Pith. sign in

REVIEW 2 major objections 7 minor 46 references

More Agent Reasoning Effort Lowers Utility in Every Test Stratum

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 06:29 UTC pith:FY3AKI2C

load-bearing objection A solid experimental-design framework for evaluating LLM coding agents, with a main finding (more reasoning effort lowers utility) that is real but narrower than it first appears. the 2 major comments →

arxiv 2607.06413 v1 pith:FY3AKI2C submitted 2026-07-07 stat.ME cs.AI

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

Hao He , Xueying Liu , Chris J. Kuhlman , Xinwei Deng This is my paper
classification stat.ME cs.AI
keywords discoverymodelagentsdatadirectionexperimentalframeworkprocess
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes treating LLM coding agents as stochastic model-discovery operators subjected to controlled factorial experiments, rather than evaluating them with single benchmark runs. The authors run two commercial coding agents (Codex and Claude Code) across 144 configurations on a networked anagram-game modeling task, varying reasoning effort, task type, optimization metric, training-data regime, and cross-validation fold. They collect eight response variables per run: two performance scores, two cost measures (dollars and time), and four process-complexity measures. The central methodological contribution is a Utility-Aligned Canonical Decomposition (UACD) that projects the multivariate effort effect onto a pre-specified utility direction balancing performance gains against cost penalties. The paper's central empirical claim is that increasing an agent's reasoning effort reliably degrades utility: the directed utility slope is negative in all eight strata and statistically significant (after Holm correction) in seven of eight. The leading canonical direction of the effort effect is utility-misaligned in all eight strata, and the effect concentrates in one dimension. In plain terms, more reasoning effort makes the agents produce longer, costlier, more complex code without a matching improvement in model quality.

Core claim

The paper establishes that for LLM coding agents acting as autonomous modelers, the reasoning-effort control functions primarily as a resource multiplier rather than a quality multiplier. Higher effort consistently increases token consumption, code length, and process complexity, but the multivariate movement it induces in the eight-dimensional response space points away from a utility direction that rewards performance and penalizes cost. This holds across both agents, both tasks (predictive and generative), and all four optimization metrics tested. The directed utility slope u^T beta_hat is negative in all eight strata and significant in seven of eight after multiple-testing correction. A

What carries the argument

The core mechanism is a two-part statistical apparatus. First, a factorial experimental design treats the agent as the object of study: six factors (agent, task, metric, reasoning effort, fold, regime) are crossed, with effort as the main treatment factor and the other design factors (fold, regime) as blocking terms, all analyzed within eight strata defined by agent-task-metric combinations. Second, a Utility-Aligned Canonical Decomposition (UACD) extracts the leading canonical direction of the effort effect from a MANOVA framework. The utility direction u = (1/2, 1/2, -1/2, -1/2, 0, 0, 0, 0)^T equally weights two performance coordinates positively and penalizes two cost coordinates. The UAC

Load-bearing premise

The utility direction u = (1/2, 1/2, -1/2, -1/2, 0, 0, 0, 0)^T assigns equal weight to two performance coordinates and equal penalty to two cost coordinates, fixing the performance-cost tradeoff a priori. The paper itself notes that a systematic sensitivity analysis over alternative utility directions is left to future work. If a different weighting — say, one that values performance far more than cost — were used, the negative slope could weaken or reverse.

What would settle it

Find a utility weighting where the directed slope u^T beta_hat becomes positive in a majority of strata, or find an agent/task/metric combination where higher reasoning effort produces a statistically significant improvement in the primary performance metric that exceeds the marginal cost increase.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reasoning-effort sliders on coding agents may need to be treated as cost controls rather than quality controls, changing how practitioners configure and budget agentic workflows.
  • The factorial-design-plus-multivariate-response methodology could be applied to other agentic AI domains (code generation, data analysis, scientific reasoning) to systematically map how agent controls move quality, cost, and process jointly.
  • The finding that agents self-assess accurately on predictive tasks but are overconfident on generative tasks suggests that deployment of agentic AI for generative modeling requires external evaluation pipelines, not just agent-internal validation.
  • If the utility-weighting is changed to value performance far more than cost, the negative directed slope could weaken or reverse, meaning the practical recommendation depends on the user's cost tolerance.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper proposes an experimental design and multivariate analysis framework for evaluating LLM coding agents as stochastic model-discovery operators. A full factorial experiment (144 runs, 140 valid) varies reasoning effort, fold, and discovery regime across two agents (Codex, Claude Code), two tasks (predictive, generative), and two primary metrics. The analysis fits a multivariate linear model per stratum, tests a pre-specified directed utility contrast (u^T beta_hat), and introduces a Utility-Aligned Canonical Decomposition (UACD) to characterize the dominant direction of the effort effect. The central empirical finding is that increasing reasoning effort lowers a pre-specified utility score (equally weighting performance and penalizing cost) in all eight strata, with significance in seven of eight after Holm correction, while performance on the primary metric remains largely flat.

Significance. The paper makes a genuine methodological contribution by treating agentic AI as an object of designed experiments rather than benchmarking. The directed utility test is a standard linear functional of the OLS estimator with proper t-intervals and Holm correction across eight strata — this is correctly specified and non-circular since u is fixed a priori (Eq. 14). The UACD provides a useful descriptive complement. The empirical finding (effort acts as a resource-control parameter, not a quality lever) is practically important and well-supported by the data. The released dataset and reproducible evaluation pipeline are a strength. The main limitation is the absence of replicates and the untested sensitivity to the utility weighting, both acknowledged by the authors.

major comments (2)
  1. §4.1, Eq. (14) and §5.2: The central directed-utility claim depends on the fixed utility direction u = (1/2, 1/2, -1/2, -1/2, 0, 0, 0, 0)^T applied to within-stratum standardized responses. The paper acknowledges that 'a systematic sensitivity analysis over alternative utility directions is left to future work.' Given that the negative slope appears driven primarily by the cost term (performance is flat per Table 4 and Figure 3), a limited sensitivity check — e.g., varying the performance-to-cost weight ratio over a small grid and reporting whether the sign of u^T beta_hat reverses — would substantially strengthen the robustness of the central claim. The stress-test analysis suggests sign reversal is unlikely but significance may attenuate under performance-heavy weightings; this is worth verifying empirically rather than leaving entirely to future work.
  2. §3.2 and §6.2: The design has no replicates (n=18 or 16 per stratum, one run per configuration). The per-stratum inference relies on the factorial structure for residual degrees of freedom. With q=6 parameters and n=18, the residual df is 12 (or 10 for a-DLD). This is adequate for the directed test but limits the stability of the 8-dimensional multivariate analysis, particularly the UACD leading direction, which the authors note is 'weakly identified within a single stratum.' The paper should more explicitly quantify the uncertainty in eta^(1) (e.g., via the bootstrap already mentioned) or at minimum report bootstrap confidence intervals for the sign, to support the sign-test claim across strata. Currently the sign test (p=0.008) assumes independent signs, which the authors flag as approximate; a brief discussion of how correlated the strata are (shared agents, tasks, folds) and whether
minor comments (7)
  1. §3.3, Eq. (6): The additive smoothing epsilon for KL6 is listed as a free parameter in the design but its value is not reported in the main text. Please report the chosen epsilon value.
  2. Appendix A: The Codex blended token rate of $6.50 per 10^6 tokens is described as a 94:6 input-to-output split. Please clarify whether this ratio was estimated from the actual runs or assumed, and whether it varies across effort levels.
  3. Figure 9, left panel: The y-axis label 'Directed utility slope u^T beta_hat' uses beta_hat notation but the text uses beta (without hat) in some places. Please standardize notation.
  4. §4.2: The sign test for eta^(1) across eight strata assumes independent signs. The authors note this is approximate. A brief remark on the expected degree of dependence would help readers calibrate the p-value.
  5. Table 4: The Bonferroni threshold alpha/4 = 0.0125 is applied to the four conditions, but the table reports both F_E and beta_L p-values. Clarify which test the correction applies to.
  6. §5.1, Figure 5: The Spearman correlations (rho=0.28 for claude-code, rho=0.01 for codex) between tokens and performance are reported without confidence intervals. Given the small sample sizes, reporting uncertainty would be useful.
  7. §3.1: The model versions (GPT-5.5, Claude Opus 4.7) are specific to the time of the experiment. A brief note on the date of the experimental runs would help readers assess generalizability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The two major comments are both well-taken and actionable. We address each below.

read point-by-point responses

  1. Referee: Major Comment 1 (§4.1, Eq. 14, §5.2): Perform a sensitivity analysis over alternative utility directions, varying the performance-to-cost weight ratio over a small grid and reporting whether the sign of u^T beta_hat reverses.

    Authors: We agree. The referee is correct that since the negative directed-utility slope is driven primarily by the cost coordinates while performance is largely flat, the robustness of the sign claim to the choice of utility weights should be verified empirically rather than deferred to future work. We will add a sensitivity analysis in the revised manuscript. Specifically, we will re-estimate u^T beta_hat across a grid of utility directions that vary the performance-to-cost weight ratio (e.g., weighting performance from 0 to 1 in increments of 0.1, with the cost weight set to 1 minus the performance weight, and process coordinates held at zero), and report for each stratum whether the sign of the directed slope reverses and whether Holm-corrected significance is maintained. Based on the referee's own assessment and our stress-test analysis, we expect the sign to remain negative across a wide range of weightings, with significance attenuating under performance-heavy weightings. We will report the empirical results honestly, including any weightings under which significance is lost. This analysis will be added to Section 5.2, and the corresponding discussion in Section 6.2 will be updated to reference the sensitivity results rather than leaving the issue entirely to future work. revision: yes

  2. Referee: Major Comment 2 (§3.2, §6.2): Quantify uncertainty in eta^(1) via bootstrap confidence intervals for the sign, and discuss how correlated the strata are (shared agents, tasks, folds) and whether the independence assumption for the sign test is reasonable.

    Authors: We agree with both parts of this comment. First, we will report bootstrap confidence intervals for eta^(1) in each stratum. The case-resampling bootstrap is already implemented and mentioned in the manuscript; we will add the resulting intervals (or at minimum, bootstrap-based intervals for the sign of eta^(1)) to Table 6 or an accompanying table, so that the sign-test claim is supported by per-stratum uncertainty quantification rather than point estimates alone. Second, we will add a brief but explicit discussion of the correlation structure across strata. The eight strata share two agents, two tasks, and three folds, so the signs of eta^(1) are not fully independent. We will discuss the nature of this dependence — for example, strata sharing the same agent may be more correlated than those that do not — and note that the sign test p-value of 0.008 should be interpreted as a lower bound on significance under positive correlation. We will also note that even under conservative assumptions about correlation (e.g., treating the four conditions as independent blocks of two strata each), the sign pattern remains unlikely under the null. This discussion will be added to Section 4.2 and/or Section 5.2, and the corresponding limitation language in Section 6.2 will be sharpened. revision: yes

Circularity Check

0 steps flagged

No circularity found. The central claim is derived from a pre-specified utility contrast applied to standard OLS estimates, with no fitted-to-prediction loop.

full rationale

The paper's central claim—that the directed utility slope u^T β̂ is negative across strata—does not reduce to its inputs by construction. The utility direction u = (1/2, 1/2, -1/2, -1/2, 0, 0, 0, 0)^T (Eq. 14, §4.1) is fixed a priori before any data analysis, not fitted from the experimental outcomes. The directed test u^T β̂_l is a standard linear functional of the OLS estimator β̂ from the multivariate linear model (Eq. 9), and its t-statistic uses the standard variance formula c_ℓℓ u^T Σ̂ u. The UACD (§4.2) uses standard MANOVA eigenstructure (Eq. 13, 15) from Pillai (1955) and Anderson (2003)—external, textbook references—to extract the leading canonical direction a_1, then measures alignment η^(1) = a_1^T u as a descriptive complement. The self-citations (He et al. 2024, 2025) concern the anagram-game application domain and prior modeling work, not the statistical methodology; they are not load-bearing for the inferential framework. The evaluation metrics (wAUC, MRI-RO, KL6, DLD) are defined from the evaluation pipeline independently of the target result. The paper explicitly acknowledges that the result depends on the choice of utility weights and leaves sensitivity analysis to future work (§5.2), which is a correctness/robustness concern, not a circularity concern. No step in the derivation chain reduces to its inputs by definition or by a fitted-then-predicted loop.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The 'model-discovery operator' is a framing concept, not a new entity. All axioms are standard statistical assumptions (normality, ordinal coding, standardization) or a single ad-hoc independence assumption for the sign test. The free parameters are utility weights (chosen a priori), a smoothing constant, a cost-imputation rate, and a validity threshold — all stated in the paper.

free parameters (4)
  • Utility direction weights u = (1/2, 1/2, -1/2, -1/2, 0, 0, 0, 0)
    Fixed a priori by the authors (Eq. 14). Equal weights on two performance coordinates, equal penalties on two cost coordinates, zero on four process coordinates. Not fitted to data, but the choice is untested for sensitivity.
  • Additive smoothing epsilon for KL6 = Not specified numerically in the main text
    Used in Eq. 6 for smoothing bin probabilities. Stated as 'identical across runs' but the value is not given in the main text.
  • Codex blended token rate = $6.50 per 10^6 tokens
    Used to impute codex dollar cost from token counts (Appendix A). Reflects a 94:6 input-to-output split at $5 and $30 per 10^6 tokens.
  • Simulation validity threshold = 90%
    Runs are kept only if at least 90% of 100-seed trajectories are valid (Appendix A). This threshold determines which runs enter the analysis.
axioms (4)
  • domain assumption LLM coding agents are stochastic model-discovery operators whose outputs are adequately characterized by a multivariate normal linear model (Eq. 9).
    Section 4.1, Eq. 9. The normality assumption is stated but not tested. With n=18 per stratum and 8 responses, the multivariate normal assumption is load-bearing for Pillai's trace F-approximation and the t-intervals.
  • domain assumption Reasoning-effort levels (low, default, max) are ordinal and comparable within an agent, coded via orthogonal polynomial contrasts (-1,0,+1) and (+1,-2,+1).
    Section 4.1. The ordinal coding assumes equal spacing between levels, which may not reflect the actual effort difference between, e.g., 'medium' and 'xhigh' for Codex.
  • domain assumption The eight response coordinates, after within-stratum standardization, are on a comparable scale for multivariate analysis.
    Section 3.3, Table 2. Standardization within stratum makes the coordinates dimensionless, but the choice of standardization (within-stratum, not across) affects the canonical directions.
  • ad hoc to paper The sign test for eta^(1) across eight strata assumes independent signs.
    Section 4.2 and Section 5.2. The paper acknowledges this is 'only approximate, because the strata share the same agents, tasks, and folds' and treats it as descriptive. The independence assumption is violated by the shared structure.

pith-pipeline@v1.1.0-glm · 23495 in / 3727 out tokens · 348773 ms · 2026-07-08T06:29:01.842514+00:00 · methodology

0 comments
read the original abstract

Large language model coding agents increasingly perform open-ended data modeling and analysis. These agents are stochastic and adaptive, and therefore their autonomous model discovery behavior cannot be adequately characterized by a single benchmark run. In this work, we propose an experimental design and analysis framework for systematically evaluating this discovery process, quantifying its variability, and identifying important factors. The proposed framework treats these agents as stochastic model-discovery operators, which map task-specific discovery data and an optimization target to a fitted model. Specifically, we investigate two such operators, Codex and Claude Code, under controlled experimental factors including agent's reasoning effort, task, optimization metric, and composition of training data. For each agent-task-metric combination, regression models and inference are conducted for multiple responses such as output quality, dollar cost, wall-clock time, and process complexity. Furthermore, we develop a utility-aligned canonical decomposition to characterize the dominant direction of the reasoning-effort effect and to assess whether that direction aligns with a performance-cost utility direction. The proposed framework is demonstrated on a testbed of networked word-forming games with insightful findings on reasoning effort with respect to cost and process complexity.

Figures

Figures reproduced from arXiv: 2607.06413 by Chris J. Kuhlman, Hao He, Xinwei Deng, Xueying Liu.

Figure 1
Figure 1. Figure 1: Illustration of the experiment and analysis pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Simplified rank-one illustration of UACD. Left panel shows the trivial case where [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance by primary metric across the reasoning-effort ladder. Each metric is [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Fresh-token use (left) and submitted-code length (right) across the reasoning-effort [PITH_FULL_IMAGE:figures/full_fig_p021_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Fresh-token use against submitted-code length (left) and against within-condition [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Primary performance under full versus partial observation, by task and metric. [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Primary metric against cost in US dollars (log axis), by condition. Color for agent [PITH_FULL_IMAGE:figures/full_fig_p024_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Agent self-assessment against the evaluated metric, one panel per condition. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Left: directed utility slope u ⊤βˆ per stratum with its model-based 95% t interval. Right: utility alignment η (1) across strata, as a boxplot over the eight point estimates. Color marks the agent. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Number of candidate models considered, one panel per reasoning-effort level. The [PITH_FULL_IMAGE:figures/full_fig_p038_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Composition of submitted model families, by task, split by agent. Task 1 is [PITH_FULL_IMAGE:figures/full_fig_p039_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages

  1. [1]

    Anderson, T. W. , title =

    work page

  2. [2]

    2025 , url =

    Aubakirova, Malika and Atallah, Alex and Clark, Chris and Summerville, Justin and Midha, Anjney , title =. 2025 , url =

    work page 2025

  3. [3]

    , title =

    Efron, Bradley and Tibshirani, Robert J. , title =

    work page

  4. [4]

    Journal of Agricultural, Biological, and Environmental Statistics , volume =

    Gertheiss, Jan , title =. Journal of Agricultural, Biological, and Environmental Statistics , volume =. 2014 , doi =

    work page 2014

  5. [5]

    Proceedings of the 2025 Winter Simulation Conference (WSC) , publisher =

    He, Hao and Liu, Xueying and Deng, Xinwei , title =. Proceedings of the 2025 Winter Simulation Conference (WSC) , publisher =. 2025 , doi =

    work page 2025

  6. [6]

    Scandinavian Journal of Statistics , volume =

    Holm, Sture , title =. Scandinavian Journal of Statistics , volume =

    work page

  7. [7]

    and Wichern, Dean W

    Johnson, Richard A. and Wichern, Dean W. , title =

    work page

  8. [8]

    , title =

    Olson, Chester L. , title =. Journal of the American Statistical Association , volume =. 1974 , doi =

    work page 1974

  9. [9]

    2026 , howpublished =

    work page 2026

  10. [10]

    Pillai, K. C. S. , title =. The Annals of Mathematical Statistics , volume =

    work page

  11. [11]

    and Christensen, William F

    Rencher, Alvin C. and Christensen, William F. , title =

    work page

  12. [12]

    Generative Modeling of Human Behavior and Social Interactions Using Abductive Analysis , booktitle =

    Ren, Yihui and Cede. Generative Modeling of Human Behavior and Social Interactions Using Abductive Analysis , booktitle =

    work page

  13. [13]

    Cognitive Science , volume =

    Sweller, John , title =. Cognitive Science , volume =. 1988 , doi =

    work page 1988

  14. [14]

    Journal of Machine Learning Research , volume =

    Flamary, R. Journal of Machine Learning Research , volume =

    work page

  15. [15]

    International Conference on Learning Representations (ICLR) , year =

    Hendrycks, Dan and Burns, Collin and Basart, Steven and Zou, Andy and Mazeika, Mantas and Song, Dawn and Steinhardt, Jacob , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  16. [16]

    Transactions on Machine Learning Research (TMLR) , year =

    Srivastava, Aarohi and Rastogi, Abhinav and Rao, Abhishek and others , title =. Transactions on Machine Learning Research (TMLR) , year =

    work page

  17. [17]

    2021 , eprint =

    Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and Edwards, Harri and Burda, Yuri and Joseph, Nicholas and Brockman, Greg and others , title =. 2021 , eprint =

    work page 2021

  18. [18]

    Transactions on Machine Learning Research (TMLR) , year =

    Liang, Percy and Bommasani, Rishi and Lee, Tony and Tsipras, Dimitris and Soylu, Dilara and Yasunaga, Michihiro and Zhang, Yian and Narayanan, Deepak and Wu, Yuhuai and Kumar, Ananya and others , title =. Transactions on Machine Learning Research (TMLR) , year =

    work page

  19. [19]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , title =

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , title =. International Conference on Learning Representations (ICLR) , year =

    work page

  20. [20]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

    work page

  21. [21]

    Proceedings of the National Academy of Sciences , volume =

    Bonabeau, Eric , title =. Proceedings of the National Academy of Sciences , volume =

    work page

  22. [22]

    and Busby, Ethan C

    Argyle, Lisa P. and Busby, Ethan C. and Fulda, Nancy and Gubler, Joshua R. and Rytting, Christopher and Wingate, David , title =. Political Analysis , volume =

    work page

  23. [23]

    and Arriaga, Rosa I

    Aher, Gati V. and Arriaga, Rosa I. and Kalai, Adam Tauman , title =. International Conference on Machine Learning (ICML) , pages =

    work page

  24. [24]

    , title =

    Horton, John J. , title =

    work page

  25. [25]

    and Cai, Carrie J

    Park, Joon Sung and O'Brien, Joseph C. and Cai, Carrie J. and Morris, Meredith Ringel and Liang, Percy and Bernstein, Michael S. , title =. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST) , year =

    work page

  26. [26]

    Humanities and Social Sciences Communications , volume =

    Gao, Chen and Lan, Xiaochong and Li, Nian and Yuan, Yuan and Ding, Jingtao and Zhou, Zhilun and Xu, Fengli and Li, Yong , title =. Humanities and Social Sciences Communications , volume =

    work page

  27. [27]

    Pawan and Dupont, Emilien and Ruiz, Francisco J

    Romera-Paredes, Bernardino and Barekatain, Mohammadamin and Novikov, Alexander and Balog, Matej and Kumar, M. Pawan and Dupont, Emilien and Ruiz, Francisco J. R. and Ellenberg, Jordan S. and Wang, Pengming and Fawzi, Omar and Kohli, Pushmeet and Fawzi, Alhussein , title =. Nature , volume =. 2024 , doi =

    work page 2024

  28. [28]

    2024 , eprint =

    Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David , title =. 2024 , eprint =

    work page 2024

  29. [29]

    and Zhu, Kehang and Horton, John J

    Manning, Benjamin S. and Zhu, Kehang and Horton, John J. , title =. 2024 , eprint =

    work page 2024

  30. [30]

    International Conference on Machine Learning (ICML) , year =

    Huang, Qian and Vora, Jian and Liang, Percy and Leskovec, Jure , title =. International Conference on Machine Learning (ICML) , year =

    work page

  31. [31]

    2024 , eprint =

    Chan, Jun Shern and Chowdhury, Neil and Jaffe, Oliver and Aung, James and Sherburn, Dane and Mays, Evan and Starace, Giulio and Liu, Kevin and others , title =. 2024 , eprint =

    work page 2024

  32. [32]

    and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =

    Yang, John and Jimenez, Carlos E. and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

    work page

  33. [33]

    , title =

    Montgomery, Douglas C. , title =

    work page

  34. [34]

    2023 , eprint =

    Chen, Lingjiao and Zaharia, Matei and Zou, James , title =. 2023 , eprint =

    work page 2023

  35. [35]

    Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) , pages =

    Mechanistic and Data-Driven Agent-Based Models to Explain Human Behavior in Online Networked Group Anagram Games , author =. Proceedings of the 2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) , pages =. 2019 , publisher =

    work page 2019

  36. [36]

    and Gonzalez, Joseph E

    Chiang, Wei-Lin and Zheng, Lianmin and Sheng, Ying and Angelopoulos, Anastasios Nikolas and Li, Tianle and Li, Dacheng and Zhu, Banghua and Zhang, Hao and Jordan, Michael I. and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Chatbot Arena: An Open Platform for Evaluating. 2024 , publisher =

    work page 2024

  37. [37]

    and Leyton-Brown, Kevin , title =

    Thornton, Chris and Hutter, Frank and Hoos, Holger H. and Leyton-Brown, Kevin , title =. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) , pages =

    work page

  38. [38]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Feurer, Matthias and Klein, Aaron and Eggensperger, Katharina and Springenberg, Jost Tobias and Blum, Manuel and Hutter, Frank , title =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

    work page

  39. [39]

    and Bartley, Nathan and Urbanowicz, Ryan J

    Olson, Randal S. and Bartley, Nathan and Urbanowicz, Ryan J. and Moore, Jason H. , title =. Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) , pages =

    work page

  40. [40]

    , title =

    Hu, Zhihao and Liu, Xueying and Deng, Xinwei and Kuhlman, Chris J. , title =. Journal of Simulation , volume =. 2024 , doi =

    work page 2024

  41. [41]

    and Deng, Xinwei , title =

    He, Hao and Liu, Xueying and Kuhlman, Chris J. and Deng, Xinwei , title =. Proceedings of the 2024 Winter Simulation Conference (WSC) , publisher =. 2024 , doi =

    work page 2024

  42. [42]

    and McNeil, Barbara J

    Hanley, James A. and McNeil, Barbara J. , title =. Radiology , volume =. 1982 , doi =

    work page 1982

  43. [43]

    and Till, Robert J

    Hand, David J. and Till, Robert J. , title =. Machine Learning , volume =. 2001 , doi =

    work page 2001

  44. [44]

    , title =

    Gneiting, Tilmann and Raftery, Adrian E. , title =. Journal of the American Statistical Association , volume =. 2007 , doi =

    work page 2007

  45. [45]

    Wu, C. F. Jeff and Hamada, Michael S. , title =. 2011 , publisher =

    work page 2011

  46. [46]

    and Fox, Emily B

    Li, Michael Y. and Fox, Emily B. and Goodman, Noah D. , title =. Proceedings of the 41st International Conference on Machine Learning , series =. 2024 , publisher =

    work page 2024