Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

Philip Torr; William Bolton

REVIEW 3 major objections 5 minor 13 references

Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frontier AI models produce plausible ideas but few expert-level picks; the paper locates the AI-scientist gap in filtering and prioritization, not generation, using F1 2026 design and MTG deck construction as test beds.

desk verdict The time-delayed benchmark framework is a real contribution, but the paper's filtering-gap conclusion outruns the evidence. read the letter →

arxiv 2608.03569 v1 pith:W7RBYHSD submitted 2026-08-04 cs.AI cs.CYcs.LGcs.MA

William Bolton , Philip Torr This is my paper

classification cs.AI cs.CYcs.LGcs.MA

keywords AI scientist benchmarkingprospective evaluationtime-delayed ground truthhypothesis generationnovelty evaluationFormula 1 engineeringMagic: The GatheringLLM reasoning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the hardest part of benchmarking AI scientists is measuring genuine novelty, because existing benchmarks use synthetic tasks or retrospective targets that models may have already seen. It proposes a different test bed: adversarial, fast-moving real-world domains in which expert practitioners independently produce observable artifacts after a known information cutoff. The authors instantiate this in two domains — 2026 Formula 1 car design, judged against 40 real pre-season innovations, and Magic: The Gathering deck construction, judged against 19 Pro Tour decklists. Across 881 F1 ideas and 108 MTG decks from six frontier models, outputs were plausible but only a minority aligned with expert solutions: the best F1 model matched 10 of 40 real innovations, and the best MTG deck recovered 5 of 7 new-set cards from a top Pro Tour deck. The paper concludes that the key capability gap for AI scientists is not idea generation but filtering, prioritization, and coherent novelty.

What carries the argument

The load-bearing mechanism is the time-delayed generation task: a fixed corpus bounded by an information cutoff, structured candidate generation, and evaluation against independently produced post-cutoff expert artifacts on two axes — outcome similarity to real innovations and intrinsic quality (citation accuracy, rule compliance, engineering plausibility). Generation runs through a three-agent pipeline separating strategic intent, regulatory analysis, and candidate production; in F1, agent 2 tags each design degree of freedom with one of five regulatory-loophole categories and cited article numbers. F1 novelty matching proceeds in three stages (dual top-k embedding retrieval, an LLM judge,

What would settle it

Re-adjudicate the 519 shortlisted F1 pairs with several domain-expert engineers using the same protocol and compare verdicts: if expert inter-rater agreement is low or the expert match count is far from the reported 55 MATCH/PARTIAL pairs, the 19/40 coverage and the filtering-gap conclusion collapse. Alternatively, pre-register the same MTG experiment on the next set release: failure to replicate the card-level Spearman rho = 0.74 would indicate the convergence signal is noise.

Watch

Extended reading notes

Core claim

The paper's central claim is that a capability-gap diagnosis can be read off a prospective-style experiment: given only pre-cutoff material (the 2026 FIA regulations; the new MTG set plus the Standard card pool), frontier LLMs generate many plausible candidates — 881 F1 ideas, 108 MTG decks — yet recover only a minority of post-cutoff expert artifacts. In F1, 19 of 40 real innovations were suggested by at least one model (best, GPT-5.2: 10); in MTG, the strongest deck recovered 5 of 7 new-set cards from a top Pro Tour deck, 15.5x the random expectation, and card-level selection tracked expert adoption (Spearman rho = 0.74, p = 0.0003). Because generation volume did not predict coverage and 8

Load-bearing premise

The F1 match counts — and the filtering-gap conclusion built on them — rest on one non-expert human reviewer's final calls on 519 pairs that an LLM judge shortlisted with an 89% false-positive rate; if those calls are unreliable, the match rates are upper bounds of unknown error.

Editorial extensions

If this is right

AI-scientist benchmarks should score filtering and prioritization explicitly, since volume-normalized coverage outperformed raw idea count and idea count did not predict matches (Spearman rho = 0.33, p = 0.52).
Pre-registering prompts, corpus, models, and runs before expert artifacts are released would make the framework fully prospective and contamination-resistant.
Scaffolding has a cost: component-focused F1 prompting widened coverage, while MTG tool-use improved deck legality but cut new-set discovery for every model (paired Wilcoxon p = 0.031).
Rule compliance is the strongest quality signal for real-world convergence (OR = 8.92, 95% CI [2.74, 29.08]), suggesting constraint-checking should be a first-class AI-scientist capability.
Any adversarial domain with periodic public reveal events and a calibratable random baseline can re-instantiate this benchmark at low marginal cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the filtering-gap thesis would be to feed each model its own generated ideas and ask it to rank the most promising: if model-side filtering does not beat random selection, the bottleneck is not merely generation-side noise.
The MTG card-level correlation (rho = 0.74) is an aggregate signal; a pre-registered replication on a future set would show whether models capture meta-level value while still missing context-dependent build-arounds, as observed with Formidable Speaker.
A multi-expert re-adjudication of the 519 F1 pairs would quantify how much of the 19/40 coverage rate is real convergence versus generous partial-match calls by a single non-expert reviewer.
The match-versus-random-baseline structure could be normalized into a general expert-convergence metric across heterogeneous domains, though the paper leaves aggregation open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the time-delayed benchmark idea is worth taking seriously, and the MTG experiment is reasonably well controlled. But the central claim—that the bottleneck is filtering rather than generation—isn't actually tested. The paper shows models over-emit and under-match; that's a low-precision observation compatible with generation failures too. To make the claim stick they'd need a direct selection/ranking experiment, or at least an F1 random baseline.

What I liked: The two-domain instantiation is smart. F1 and MTG have different structure, different verification methods. MTG uses deterministic decklist comparisons with hypergeometric baselines and Monte Carlo bounds; that's solid. The card-level Spearman correlation (rho=0.74) is a real result. The F1 side has a three-agent pipeline, quality filters, and Fisher tests; the rule-compliance odds ratio is suggestive. They also ship code, data, and prompts. That's reproducible, which puts them ahead of many AI-for-science papers. The limitations section is honest about proxy domains and non-prospective timing.

Where it's soft: The F1 match judgments rest on one author who isn't a domain expert, and the LLM judge's labels were 89% false positives. That means the 19/40 and 10/40 numbers are upper bounds with unknown error. There's no random baseline for F1 either, so we don't know how many matches a chance generator would get. And the filtering conclusion—the headline—is an interpretation. The union coverage (19/40) shows generation isn't absent, but it doesn't show models could have selected those ideas if asked. They never test that.

The MTG side is less affected; the random baseline at least gives an anchor. But even there, the over-prediction patterns could reflect imprecise knowledge about card strength, not a filtering failure per se.

Overall: It's a worthwhile paper, worth a serious referee, but it needs a revision that either strengthens the adjudication and baselines or softens the central claim. I'd bring it to a reading group and cite it if I were working on AI scientist evaluation.

Referee Report

3 major / 5 minor

Summary. The paper proposes a general framework for benchmarking AI scientist capabilities using adversarial, fast-moving real-world domains with public, independently produced expert artifacts as time-delayed ground truth. It instantiates the framework in two domains: F1 2026 technical regulations (free-form car design ideation against 40 real pre-season innovations) and MTG Lorwyn Eclipsed (deck construction against 19 Pro Tour decklists). Six frontier LLMs are run through a three-agent pipeline, and outputs are evaluated for overlap with expert artifacts and intrinsic quality. The main empirical findings are that models produce many plausible outputs but few match expert solutions (best F1 model GPT-5.2 matched 10/40 real innovations; best MTG deck recovered 5/7 new-set cards from one Pro Tour deck), that aggregate MTG card popularity correlates with Pro Tour adoption (Spearman rho = 0.74), and that quality signals such as rule compliance are associated with F1 matches. The paper concludes that the key capability gap is not idea generation but filtering, prioritization, and coherent novelty.

Significance. If the framework is valid, it offers a useful complement to existing retrospective and synthetic benchmarks, with the important property that ground-truth artifacts post-date model training cutoffs and are produced independently by experts. The MTG side is methodologically strong: deterministic decklist comparisons, explicit hypergeometric random baselines with Monte Carlo bounds, and a card-level rank correlation are clearly specified. The F1 side has a genuinely interesting setup: free-form ideation under a large regulatory corpus judged against a curated set of real innovations. The paper also provides code, data, and prompts, and is transparent about the noise in its evaluation pipeline. The central conclusion about a filtering gap is an interpretive claim that, while plausible and partially supported, is not directly tested; this is the main weakness. The framework itself is portable and likely to be of interest to the AI-scientist benchmarking community.

major comments (3)

[Abstract and Section 6 (Discussion)] The central claim that the key capability gap is 'not idea generation, but filtering, prioritization, and coherent novelty' is not directly tested anywhere in the paper. Low match rates plus high generation volume are consistent with both (a) a filtering/prioritization failure and (b) a generation/understanding failure in which models cannot produce ideas sufficiently aligned with real-world constraints. The union-coverage result (19/40 F1 innovations covered by at least one model) is suggestive but does not isolate filtering, since no single model reliably produces those ideas and no experiment asks models to rank or select from a candidate pool. The paper should either add a direct selection/ranking test (e.g., present the union of generated ideas to a model and ask it to prioritize, then measure whether the matched ideas are ranked above the rest) or substantially soften the attributi
[Section 4.2 and Supplementary Methods: Human review protocol] All F1 match counts and the Fisher exact tests in Section 5.1 depend on final labels assigned by a single human reviewer who, by the paper's own statement, is not a domain expert in F1 engineering or competitive Magic. The upstream LLM judge had an 89% false-positive rate, and embedding cosine similarity did not discriminate matched from unmatched pairs within the shortlist. Under these conditions, the reliability of the 19/40 and per-model counts is unknown. The manuscript should report inter-rater reliability (e.g., a second reviewer on a subsample), provide a domain-expert audit of at least the MATCH/PARTIAL calls, or make the adjudication criteria machine-checkable. Without this, the headline F1 numbers are upper bounds of unknown error.
[Section 5.1 and Section 4.2] Unlike the MTG domain, the F1 evaluation has no random baseline or negative control. The paper reports that 19/40 real innovations were recovered by at least one model and that GPT-5.2 matched 10/40, but it never estimates how many matches would be expected by chance under a comparable generation process (e.g., generic regulatory-grounded ideas). This matters because the F1 ideas are high-level conceptual descriptions and the ground-truth innovations are described in similar public language; chance overlap may be non-trivial. The absence of a baseline undermines the interpretation of 'few align with real-world expert solutions.' A simple control could be to generate ideas with an untrained or instruction-oblivious model, or to shuffle/perturb the real innovation descriptions and recompute the matching pipeline.

minor comments (5)

[Section 3.1.1] 'FIA 2026 Section CTechnical Regulations' appears to have a missing space: should be 'Section C Technical Regulations'.
[Section 5.1] The sentence 'Qwen3 235B thinking partially matching 3 real-world innovations' is ungrammatical and ambiguous; it likely means 'The Qwen3 235B (thinking) model partially matched 3 real-world innovations.'
[Supplementary Methods, Run structure] For MTG, the text says each model under each pipeline configuration was run three times, yielding 18 generated decks per model per configuration. Earlier in Section 5.2, 108 generated decks are reported across six models. Please clarify the arithmetic (six models x two configurations x nine decks) and ensure the deck counts are consistent.
[Supplementary Methods, F1 loophole tagging] Typo: 'likley' should be 'likely' in the sentence 'This gain, however, likley comes from differential prompting...'.
[Section 4.2] The random baseline uses N = 261 non-land new-set cards, but the set is described as containing 267 unique cards. Please explain the six excluded cards (presumably basic lands or other non-deck-playable cards) explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are empirical benchmark results evaluated against externally produced expert artifacts; no fitted parameter, definitional equivalence, or load-bearing self-citation forces the conclusion.

full rationale

The paper's central claim is an empirical one: frontier models produce plausible but largely misaligned outputs in two adversarial real-world domains, and the authors interpret this as a filtering/prioritization gap rather than a pure generation gap. This is not a derivation that assumes its own result. In F1, the ground truth is a curated set of 40 real pre-season innovations sourced from public technical analyses published after all model knowledge cutoffs; matching is determined by a two-stage pipeline (LLM shortlisting plus human review) that compares physical effects and geometry. In MTG, evaluation is deterministic decklist overlap against 19 Pro Tour decklists, with explicit hypergeometric random baselines (E[overlap] = kK/N and E_agg) used as external anchors rather than as fitted parameters. No equation in the paper rewrites its output as an input; no parameter is fitted to a subset of data and then reported as a prediction. The reported correlations (e.g., Spearman rho = 0.74 between generated-card frequency and PT adoption) are straightforward empirical associations, not self-consistent constructions. The paper's own discussion and supplementary methods openly quantify adjudication noise (89% of LLM-judge-flagged pairs downgraded) and list limitations, which is self-critical rather than circular. Concerns about whether the filtering-gap conclusion is the only possible interpretation of low precision, or about reviewer reliability, are validity questions, not circularity. Therefore no specific circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of the ground-truth artifacts (F1 innovations from public analyses, PT decklists), the reliability of the human/LLM match adjudication, and the assumption of clean information cutoffs. The MTG baselines are mathematically sound; the F1 matching pipeline is the least externally anchored component.

free parameters (1)

dual top-k retrieval shortlist size k = 3
Hand-chosen number of nearest real innovations per generated idea (and vice versa) used to build the F1 human-review pool; no sensitivity analysis is reported, and match counts could shift with k.

assumptions (4)

domain assumption The 40 curated F1 innovations are an unbiased and complete sample of real 2026 pre-season design innovations.
Ground truth compiled from five YouTube videos and four articles with heavy emphasis on visible aero and packaging; the paper itself notes decisive confidential performance innovations are missing.
domain assumption A single non-expert human reviewer can reliably adjudicate MATCH/PARTIAL/NO MATCH after an LLM judge with an 89% false-positive rate.
Human review protocol in Supplementary Methods; all F1 match statistics depend on this noisy two-stage pipeline.
domain assumption The models' training cutoffs guarantee the input corpora (FIA 2026 regulations, Lorwyn Eclipsed cards) were unseen in training.
Reported cutoffs all predate the document releases, but the paper does not empirically rule out earlier draft regulations or card spoilers.
standard math Hypergeometric random baseline correctly models chance overlap in MTG deck comparisons.
E[overlap]=kK/N and E_agg are standard combinatorial expectations under uniform independent draws; the paper argues correlation makes E_agg an upper bound on random rediscovery.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities." pith.science (2026). https://pith.science/paper/W7RBYHSD

@misc{pith2026260803569,
  author       = {Pith},
  title        = {Pith review of: Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7RBYHSD}},
  note         = {Machine review of arXiv:2608.03569}
}

abstract

Benchmarking the ability of AI scientists to generate novel ideas is notoriously difficult. Existing benchmarks in this field have made progress in evaluating scientific reasoning and research replication, but often rely on synthetic tasks or retrospective targets, which may be confounded by prior exposure. We hypothesize that complex, adversarial, fast-moving real-world domains where expert practitioners independently generate observable outputs can provide a practical solution to fill this gap and evaluate the capabilities needed for AI scientists, including reasoning, novelty, and hypothesis formulation. We instantiate this framework in two structurally different domains, Formula 1 (F1), where models ideate around car design concepts for the 2026 season, and real pre-season innovations provide a ground truth, and Magic: The Gathering (MTG), where models propose decks from a recently updated card pool and are evaluated against 19 Pro Tour (PT) decklists. Across both domains, models produce plausible outputs, but few align with real-world expert solutions. In F1, the best model, GPT-5.2 matched 10 of 40 real innovations with 166 ideas proposed across runs. In MTG, the best deck from Gemini 3 Flash recovered 5 of 7 new-set cards from the third-place PT deck, and across all 108 decks, the cards models selected most often were also the cards most widely adopted by PT decks (Spearman $\rho = 0.74$, $p = 0.0003$). These results suggest that a key capability gap for AI scientists is not idea generation, but filtering, prioritization, and coherent novelty.

Figures

Figures reproduced from arXiv: 2608.03569 by the authors.

**Figure 1.** Human-confirmed matches between model-generated ideas and real 2026 F1 innovations. Rows show the real innovations, grouped by car area, and columns show each model under general (full-car) and component-focused configurations. Green cells denote matches, yellow cells denote partial matches, with cell values indicating the number of generated ideas that matched [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Fraction of new cards from each generated deck’s closest PT match that were correctly predicted, broken down by model and pipeline configuration. Each marker is one generated deck. The grey band marks the per-pair random expectation; points above it indicate above-chance recovery [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Per-model coverage of the 19 new cards present in any of the PT decks. Rows are PT new cards, ordered by the number of ground-truth decks containing them (right bar). Both one-shot and tool-use configurations are shown for each model. Cell values give the number of generated decks (out of 9) that included the card. the upper quartile of its random-baseline band in both configurations (8 of 19 one-shot, 1.34× expect… view at source ↗

Figures from the paper (8 more)

**Figure 4.** Figure 4: Cross-run idea convergence by model. For each idea in run i the highest cosine similarity to any idea in run j ̸= i is computed using lay summary embeddings. Box plots show the distribution of these best-match similarities pooled across all run pairs for each model’s t…

**Figure 5.** Figure 5: Match rate by LLM-judge compliance verdict. The left panel shows the pooled rate across all six models. The right panel shows per-model rates broken down by compliance verdict [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Real-innovation coverage by configuration. For each model the blue bars show distinct real innovations matched via three independent general (full-car) prompt runs, the red bars via the component-focused prompt set (one run per car area), and the diamond marker denotes…

**Figure 7.** Figure 7: Distribution of matched and unmatched real innovations by car region. Each row is one of the eight regions used by both the model-generated and real-innovation schemas. Green segments count innovations that any of the six models matched (MATCH or PARTIAL after human re…

**Figure 8.** Figure 8: Aggregate count of unique new cards found by each configuration (out of 19). Black whiskers overlay the configuration-matched random-baseline 25–75 percentile range estimated by Monte Carlo; a bar whose top exceeds the upper whisker sits above the 75th percentile of th…

**Figure 9.** Figure 9: Deterministic legality of generated decks by model and pipeline configuration. Left: decks (out of 9) with a valid 60-card main deck. Right: decks with a valid 15-card sideboard. Tool-use scaffolding substantially increases main-deck validity for GPT-5.2 (11% → 100%) a…

**Figure 10.** Figure 10: Every generated PT deck pair, with non-land card overlap on the x-axis and new-card overlap on the y-axis. Filled markers are one-shot generations, hollow markers are tool-use, colored by model. Points are jittered for visibility [PITH_FULL_IMAGE:figures/full_fig_p01…

**Figure 11.** Figure 11: Within-configuration spread of generated-deck quality. For each generated deck we record its maximum non-land card overlap with any PT deck. Each dot is one of the 9 decks per configuration. Boxes show the inter-quartile range, whiskers the min/max, and the central ba…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages

[1]

Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

Chen, Z., Chen, S., Ning, Y ., Zhang, Q., Wang, B., Yu, B., Li, Y ., Liao, Z., Wei, C., Lu, Z., et al. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

arXiv
[4]

Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. Forecastbench: A dy- namic benchmark of ai forecasting capabilities.arXiv preprint arXiv:2409.19839,

arXiv
[6]

T., Foerster, J., Clune, J., and Ha, D

Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

arXiv
[7]

P., Surana, H., Agarwal, D., Mishra, B

Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., V ora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725,

arXiv
[8]

C., Barabasi, D

Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadol- ski, T., Sulovari, A., Landsness, E. C., Barabasi, D. L., Narayanan, S., Evans, N., et al. Kosmos: An ai scientist for autonomous discovery.arXiv preprint arXiv:2511.02824,

arXiv
[9]

Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

arXiv
[11]

The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API

8 Adversarial Fast-Moving Real-World Domains for Evaluating AI Scientists Supplementary Methods Models Six frontier LLMs spanning three providers were evaluated. The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API. The Google models Gemini 3 Flash and Gemini 3.1 Pro (both January 20...

work page 2025
[12]

F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel

and Lorwyn Eclipsed (released 23 January 2026), so neither corpus could have appeared in any model’s training data. F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel. Ground-truth collection The reference set of 40 real 2026 pre-season innovations was compiled from publicly available technical analysis publish...

work page 2026

Show all 13 references

[15]

The Top 15 by Day 2 standings and four featured archetype-diverse builds were collected from official PT Lorwyn Eclipsed coverage (30 January – 1 February 2026)

15 magic.gg/events/pro-tour-lorwyn- eclipsed PT Lorwyn Eclipsed (featured archetype- diverse builds) 4 magic.gg/events/pro-tour-lorwyn- eclipsed Total 19 Table 3.Provenance of the 19 PT Lorwyn Eclipsed reference decks. The Top 15 by Day 2 standings and four featured archetype-...

2026
[2015]

S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al

Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

arXiv
[2021]

Hypobench: Towards systematic and principled benchmarking for hy- pothesis generation.arXiv preprint arXiv:2504.11524,

7 Adversarial Fast-Moving Real-World Domains for Evaluating AI Scientists Liu, H., Huang, S., Hu, J., Zhou, Y ., and Tan, C. Hypobench: Towards systematic and principled benchmarking for hy- pothesis generation.arXiv preprint arXiv:2504.11524,
[2024]

The need for verification in ai-driven scientific discov- ery.arXiv preprint arXiv:2509.01398,

Cornelio, C., Ito, T., Cory-Wright, R., Dash, S., and Horesh, L. The need for verification in ai-driven scientific discov- ery.arXiv preprint arXiv:2509.01398,
[2025]

com/DGoettlich/history-llms

URL https://github. com/DGoettlich/history-llms. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

Chen, Z., Chen, S., Ning, Y ., Zhang, Q., Wang, B., Yu, B., Li, Y ., Liao, Z., Wei, C., Lu, Z., et al. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

arXiv

[2] [4]

Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. Forecastbench: A dy- namic benchmark of ai forecasting capabilities.arXiv preprint arXiv:2409.19839,

arXiv

[3] [6]

T., Foerster, J., Clune, J., and Ha, D

Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

arXiv

[4] [7]

P., Surana, H., Agarwal, D., Mishra, B

Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., V ora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725,

arXiv

[5] [8]

C., Barabasi, D

Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadol- ski, T., Sulovari, A., Landsness, E. C., Barabasi, D. L., Narayanan, S., Evans, N., et al. Kosmos: An ai scientist for autonomous discovery.arXiv preprint arXiv:2511.02824,

arXiv

[6] [9]

Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

arXiv

[7] [11]

The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API

8 Adversarial Fast-Moving Real-World Domains for Evaluating AI Scientists Supplementary Methods Models Six frontier LLMs spanning three providers were evaluated. The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API. The Google models Gemini 3 Flash and Gemini 3.1 Pro (both January 20...

work page 2025

[8] [12]

F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel

and Lorwyn Eclipsed (released 23 January 2026), so neither corpus could have appeared in any model’s training data. F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel. Ground-truth collection The reference set of 40 real 2026 pre-season innovations was compiled from publicly available technical analysis publish...

work page 2026