Pith. sign in

REVIEW 3 major objections 5 minor 13 references

Frontier AI models produce plausible ideas but few expert-level picks; the paper locates the AI-scientist gap in filtering and prioritization, not generation, using F1 2026 design and MTG deck construction as test beds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

In two adversarial real-world domains with time-delayed expert ground truth, frontier LLMs over-generate plausible ideas but under-select the specific solutions experts adopted, indicating a filtering and prioritization gap.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The time-delayed benchmark framework is a real contribution, but the paper's filtering-gap conclusion outruns the evidence. the 3 major comments →

arxiv 2608.03569 v1 pith:W7RBYHSD submitted 2026-08-04 cs.AI cs.CYcs.LGcs.MA

Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities

William Bolton , Philip Torr This is my paper
classification cs.AI cs.CYcs.LGcs.MA
keywords AI scientist benchmarkingprospective evaluationtime-delayed ground truthhypothesis generationnovelty evaluationFormula 1 engineeringMagic: The GatheringLLM reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the hardest part of benchmarking AI scientists is measuring genuine novelty, because existing benchmarks use synthetic tasks or retrospective targets that models may have already seen. It proposes a different test bed: adversarial, fast-moving real-world domains in which expert practitioners independently produce observable artifacts after a known information cutoff. The authors instantiate this in two domains — 2026 Formula 1 car design, judged against 40 real pre-season innovations, and Magic: The Gathering deck construction, judged against 19 Pro Tour decklists. Across 881 F1 ideas and 108 MTG decks from six frontier models, outputs were plausible but only a minority aligned with expert solutions: the best F1 model matched 10 of 40 real innovations, and the best MTG deck recovered 5 of 7 new-set cards from a top Pro Tour deck. The paper concludes that the key capability gap for AI scientists is not idea generation but filtering, prioritization, and coherent novelty.

Core claim

The paper's central claim is that a capability-gap diagnosis can be read off a prospective-style experiment: given only pre-cutoff material (the 2026 FIA regulations; the new MTG set plus the Standard card pool), frontier LLMs generate many plausible candidates — 881 F1 ideas, 108 MTG decks — yet recover only a minority of post-cutoff expert artifacts. In F1, 19 of 40 real innovations were suggested by at least one model (best, GPT-5.2: 10); in MTG, the strongest deck recovered 5 of 7 new-set cards from a top Pro Tour deck, 15.5x the random expectation, and card-level selection tracked expert adoption (Spearman rho = 0.74, p = 0.0003). Because generation volume did not predict coverage and 8

What carries the argument

The load-bearing mechanism is the time-delayed generation task: a fixed corpus bounded by an information cutoff, structured candidate generation, and evaluation against independently produced post-cutoff expert artifacts on two axes — outcome similarity to real innovations and intrinsic quality (citation accuracy, rule compliance, engineering plausibility). Generation runs through a three-agent pipeline separating strategic intent, regulatory analysis, and candidate production; in F1, agent 2 tags each design degree of freedom with one of five regulatory-loophole categories and cited article numbers. F1 novelty matching proceeds in three stages (dual top-k embedding retrieval, an LLM judge,

Load-bearing premise

The F1 match counts — and the filtering-gap conclusion built on them — rest on one non-expert human reviewer's final calls on 519 pairs that an LLM judge shortlisted with an 89% false-positive rate; if those calls are unreliable, the match rates are upper bounds of unknown error.

What would settle it

Re-adjudicate the 519 shortlisted F1 pairs with several domain-expert engineers using the same protocol and compare verdicts: if expert inter-rater agreement is low or the expert match count is far from the reported 55 MATCH/PARTIAL pairs, the 19/40 coverage and the filtering-gap conclusion collapse. Alternatively, pre-register the same MTG experiment on the next set release: failure to replicate the card-level Spearman rho = 0.74 would indicate the convergence signal is noise.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • AI-scientist benchmarks should score filtering and prioritization explicitly, since volume-normalized coverage outperformed raw idea count and idea count did not predict matches (Spearman rho = 0.33, p = 0.52).
  • Pre-registering prompts, corpus, models, and runs before expert artifacts are released would make the framework fully prospective and contamination-resistant.
  • Scaffolding has a cost: component-focused F1 prompting widened coverage, while MTG tool-use improved deck legality but cut new-set discovery for every model (paired Wilcoxon p = 0.031).
  • Rule compliance is the strongest quality signal for real-world convergence (OR = 8.92, 95% CI [2.74, 29.08]), suggesting constraint-checking should be a first-class AI-scientist capability.
  • Any adversarial domain with periodic public reveal events and a calibratable random baseline can re-instantiate this benchmark at low marginal cost.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the filtering-gap thesis would be to feed each model its own generated ideas and ask it to rank the most promising: if model-side filtering does not beat random selection, the bottleneck is not merely generation-side noise.
  • The MTG card-level correlation (rho = 0.74) is an aggregate signal; a pre-registered replication on a future set would show whether models capture meta-level value while still missing context-dependent build-arounds, as observed with Formidable Speaker.
  • A multi-expert re-adjudication of the 519 F1 pairs would quantify how much of the 19/40 coverage rate is real convergence versus generous partial-match calls by a single non-expert reviewer.
  • The match-versus-random-baseline structure could be normalized into a general expert-convergence metric across heterogeneous domains, though the paper leaves aggregation open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a general framework for benchmarking AI scientist capabilities using adversarial, fast-moving real-world domains with public, independently produced expert artifacts as time-delayed ground truth. It instantiates the framework in two domains: F1 2026 technical regulations (free-form car design ideation against 40 real pre-season innovations) and MTG Lorwyn Eclipsed (deck construction against 19 Pro Tour decklists). Six frontier LLMs are run through a three-agent pipeline, and outputs are evaluated for overlap with expert artifacts and intrinsic quality. The main empirical findings are that models produce many plausible outputs but few match expert solutions (best F1 model GPT-5.2 matched 10/40 real innovations; best MTG deck recovered 5/7 new-set cards from one Pro Tour deck), that aggregate MTG card popularity correlates with Pro Tour adoption (Spearman rho = 0.74), and that quality signals such as rule compliance are associated with F1 matches. The paper concludes that the key capability gap is not idea generation but filtering, prioritization, and coherent novelty.

Significance. If the framework is valid, it offers a useful complement to existing retrospective and synthetic benchmarks, with the important property that ground-truth artifacts post-date model training cutoffs and are produced independently by experts. The MTG side is methodologically strong: deterministic decklist comparisons, explicit hypergeometric random baselines with Monte Carlo bounds, and a card-level rank correlation are clearly specified. The F1 side has a genuinely interesting setup: free-form ideation under a large regulatory corpus judged against a curated set of real innovations. The paper also provides code, data, and prompts, and is transparent about the noise in its evaluation pipeline. The central conclusion about a filtering gap is an interpretive claim that, while plausible and partially supported, is not directly tested; this is the main weakness. The framework itself is portable and likely to be of interest to the AI-scientist benchmarking community.

major comments (3)
  1. [Abstract and Section 6 (Discussion)] The central claim that the key capability gap is 'not idea generation, but filtering, prioritization, and coherent novelty' is not directly tested anywhere in the paper. Low match rates plus high generation volume are consistent with both (a) a filtering/prioritization failure and (b) a generation/understanding failure in which models cannot produce ideas sufficiently aligned with real-world constraints. The union-coverage result (19/40 F1 innovations covered by at least one model) is suggestive but does not isolate filtering, since no single model reliably produces those ideas and no experiment asks models to rank or select from a candidate pool. The paper should either add a direct selection/ranking test (e.g., present the union of generated ideas to a model and ask it to prioritize, then measure whether the matched ideas are ranked above the rest) or substantially soften the attributi
  2. [Section 4.2 and Supplementary Methods: Human review protocol] All F1 match counts and the Fisher exact tests in Section 5.1 depend on final labels assigned by a single human reviewer who, by the paper's own statement, is not a domain expert in F1 engineering or competitive Magic. The upstream LLM judge had an 89% false-positive rate, and embedding cosine similarity did not discriminate matched from unmatched pairs within the shortlist. Under these conditions, the reliability of the 19/40 and per-model counts is unknown. The manuscript should report inter-rater reliability (e.g., a second reviewer on a subsample), provide a domain-expert audit of at least the MATCH/PARTIAL calls, or make the adjudication criteria machine-checkable. Without this, the headline F1 numbers are upper bounds of unknown error.
  3. [Section 5.1 and Section 4.2] Unlike the MTG domain, the F1 evaluation has no random baseline or negative control. The paper reports that 19/40 real innovations were recovered by at least one model and that GPT-5.2 matched 10/40, but it never estimates how many matches would be expected by chance under a comparable generation process (e.g., generic regulatory-grounded ideas). This matters because the F1 ideas are high-level conceptual descriptions and the ground-truth innovations are described in similar public language; chance overlap may be non-trivial. The absence of a baseline undermines the interpretation of 'few align with real-world expert solutions.' A simple control could be to generate ideas with an untrained or instruction-oblivious model, or to shuffle/perturb the real innovation descriptions and recompute the matching pipeline.
minor comments (5)
  1. [Section 3.1.1] 'FIA 2026 Section CTechnical Regulations' appears to have a missing space: should be 'Section C Technical Regulations'.
  2. [Section 5.1] The sentence 'Qwen3 235B thinking partially matching 3 real-world innovations' is ungrammatical and ambiguous; it likely means 'The Qwen3 235B (thinking) model partially matched 3 real-world innovations.'
  3. [Supplementary Methods, Run structure] For MTG, the text says each model under each pipeline configuration was run three times, yielding 18 generated decks per model per configuration. Earlier in Section 5.2, 108 generated decks are reported across six models. Please clarify the arithmetic (six models x two configurations x nine decks) and ensure the deck counts are consistent.
  4. [Supplementary Methods, F1 loophole tagging] Typo: 'likley' should be 'likely' in the sentence 'This gain, however, likley comes from differential prompting...'.
  5. [Section 4.2] The random baseline uses N = 261 non-land new-set cards, but the set is described as containing 267 unique cards. Please explain the six excluded cards (presumably basic lands or other non-deck-playable cards) explicitly.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical benchmark results evaluated against externally produced expert artifacts; no fitted parameter, definitional equivalence, or load-bearing self-citation forces the conclusion.

full rationale

The paper's central claim is an empirical one: frontier models produce plausible but largely misaligned outputs in two adversarial real-world domains, and the authors interpret this as a filtering/prioritization gap rather than a pure generation gap. This is not a derivation that assumes its own result. In F1, the ground truth is a curated set of 40 real pre-season innovations sourced from public technical analyses published after all model knowledge cutoffs; matching is determined by a two-stage pipeline (LLM shortlisting plus human review) that compares physical effects and geometry. In MTG, evaluation is deterministic decklist overlap against 19 Pro Tour decklists, with explicit hypergeometric random baselines (E[overlap] = kK/N and E_agg) used as external anchors rather than as fitted parameters. No equation in the paper rewrites its output as an input; no parameter is fitted to a subset of data and then reported as a prediction. The reported correlations (e.g., Spearman rho = 0.74 between generated-card frequency and PT adoption) are straightforward empirical associations, not self-consistent constructions. The paper's own discussion and supplementary methods openly quantify adjudication noise (89% of LLM-judge-flagged pairs downgraded) and list limitations, which is self-critical rather than circular. Concerns about whether the filtering-gap conclusion is the only possible interpretation of low precision, or about reviewer reliability, are validity questions, not circularity. Therefore no specific circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 0 invented entities

The central claims rest on the validity of the ground-truth artifacts (F1 innovations from public analyses, PT decklists), the reliability of the human/LLM match adjudication, and the assumption of clean information cutoffs. The MTG baselines are mathematically sound; the F1 matching pipeline is the least externally anchored component.

free parameters (1)
  • dual top-k retrieval shortlist size k = 3
    Hand-chosen number of nearest real innovations per generated idea (and vice versa) used to build the F1 human-review pool; no sensitivity analysis is reported, and match counts could shift with k.
axioms (4)
  • domain assumption The 40 curated F1 innovations are an unbiased and complete sample of real 2026 pre-season design innovations.
    Ground truth compiled from five YouTube videos and four articles with heavy emphasis on visible aero and packaging; the paper itself notes decisive confidential performance innovations are missing.
  • domain assumption A single non-expert human reviewer can reliably adjudicate MATCH/PARTIAL/NO MATCH after an LLM judge with an 89% false-positive rate.
    Human review protocol in Supplementary Methods; all F1 match statistics depend on this noisy two-stage pipeline.
  • domain assumption The models' training cutoffs guarantee the input corpora (FIA 2026 regulations, Lorwyn Eclipsed cards) were unseen in training.
    Reported cutoffs all predate the document releases, but the paper does not empirically rule out earlier draft regulations or card spoilers.
  • standard math Hypergeometric random baseline correctly models chance overlap in MTG deck comparisons.
    E[overlap]=kK/N and E_agg are standard combinatorial expectations under uniform independent draws; the paper argues correlation makes E_agg an upper bound on random rediscovery.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities." pith.science (2026). https://pith.science/paper/W7RBYHSD

@misc{pith2026260803569,
  author       = {Pith},
  title        = {Pith review of: Adversarial Fast-Moving Real-World Domains as Test Beds for Benchmarking AI Scientist Capabilities},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W7RBYHSD}},
  note         = {Machine review of arXiv:2608.03569}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Benchmarking the ability of AI scientists to generate novel ideas is notoriously difficult. Existing benchmarks in this field have made progress in evaluating scientific reasoning and research replication, but often rely on synthetic tasks or retrospective targets, which may be confounded by prior exposure. We hypothesize that complex, adversarial, fast-moving real-world domains where expert practitioners independently generate observable outputs can provide a practical solution to fill this gap and evaluate the capabilities needed for AI scientists, including reasoning, novelty, and hypothesis formulation. We instantiate this framework in two structurally different domains, Formula 1 (F1), where models ideate around car design concepts for the 2026 season, and real pre-season innovations provide a ground truth, and Magic: The Gathering (MTG), where models propose decks from a recently updated card pool and are evaluated against 19 Pro Tour (PT) decklists. Across both domains, models produce plausible outputs, but few align with real-world expert solutions. In F1, the best model, GPT-5.2 matched 10 of 40 real innovations with 166 ideas proposed across runs. In MTG, the best deck from Gemini 3 Flash recovered 5 of 7 new-set cards from the third-place PT deck, and across all 108 decks, the cards models selected most often were also the cards most widely adopted by PT decks (Spearman $\rho = 0.74$, $p = 0.0003$). These results suggest that a key capability gap for AI scientists is not idea generation, but filtering, prioritization, and coherent novelty.

Figures

Figures reproduced from arXiv: 2608.03569 by Philip Torr, William Bolton.

Figure 1
Figure 1. Figure 1: Human-confirmed matches between model-generated ideas and real 2026 F1 innovations. Rows show the real innova￾tions, grouped by car area, and columns show each model under general (full-car) and component-focused configurations. Green cells denote matches, yellow cells denote partial matches, with cell values indicating the number of generated ideas that matched [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Fraction of new cards from each generated deck’s closest PT match that were correctly predicted, broken down by model and pipeline configuration. Each marker is one generated deck. The grey band marks the per-pair random expectation; points above it indicate above-chance recovery [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Per-model coverage of the 19 new cards present in any of the PT decks. Rows are PT new cards, ordered by the number of ground-truth decks containing them (right bar). Both one-shot and tool-use configurations are shown for each model. Cell values give the number of generated decks (out of 9) that included the card. the upper quartile of its random-baseline band in both config￾urations (8 of 19 one-shot, 1.… view at source ↗
Figure 4
Figure 4. Figure 4: Cross-run idea convergence by model. For each idea in run i the highest cosine similarity to any idea in run j ̸= i is computed using lay summary embeddings. Box plots show the distribution of these best-match similarities pooled across all run pairs for each model’s three independent general runs, with white diamonds denoting the means [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Match rate by LLM-judge compliance verdict. The left panel shows the pooled rate across all six models. The right panel shows per-model rates broken down by compliance verdict [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Real-innovation coverage by configuration. For each model the blue bars show distinct real innovations matched via three independent general (full-car) prompt runs, the red bars via the component-focused prompt set (one run per car area), and the diamond marker denotes the union across both configurations. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of matched and unmatched real innovations by car region. Each row is one of the eight regions used by both the model-generated and real-innovation schemas. Green segments count innovations that any of the six models matched (MATCH or PARTIAL after human review), grey segments count innovations no model matched. MTG Tool-use scaffolding affects main-deck legality very differently across models.… view at source ↗
Figure 8
Figure 8. Figure 8: Aggregate count of unique new cards found by each configuration (out of 19). Black whiskers overlay the configuration-matched random-baseline 25–75 percentile range estimated by Monte Carlo; a bar whose top exceeds the upper whisker sits above the 75th percentile of that configuration’s random baseline. Gemini 3 Flash is the only model to do so in both configurations (8/19 one-shot, 1.34× expected; 3/19 to… view at source ↗
Figure 9
Figure 9. Figure 9: Deterministic legality of generated decks by model and pipeline configuration. Left: decks (out of 9) with a valid 60-card main deck. Right: decks with a valid 15-card sideboard. Tool-use scaffolding substantially increases main-deck validity for GPT-5.2 (11% → 100%) and Qwen3 (0% → 89%), but no statistical difference is observed across all six models (Friedman p = 0.24) [PITH_FULL_IMAGE:figures/full_fig_… view at source ↗
Figure 10
Figure 10. Figure 10: Every generated PT deck pair, with non-land card overlap on the x-axis and new-card overlap on the y-axis. Filled markers are one-shot generations, hollow markers are tool-use, colored by model. Points are jittered for visibility [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Within-configuration spread of generated-deck quality. For each generated deck we record its maximum non-land card overlap with any PT deck. Each dot is one of the 9 decks per configuration. Boxes show the inter-quartile range, whiskers the min/max, and the central bar the median. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 3 canonical work pages

  1. [1]

    Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

    Chen, Z., Chen, S., Ning, Y ., Zhang, Q., Wang, B., Yu, B., Li, Y ., Liao, Z., Wei, C., Lu, Z., et al. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery.arXiv preprint arXiv:2410.05080,

    Pith/arXiv arXiv

  2. [4]

    Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. Forecastbench: A dy- namic benchmark of ai forecasting capabilities.arXiv preprint arXiv:2409.19839,

    Pith/arXiv arXiv

  3. [6]

    T., Foerster, J., Clune, J., and Ha, D

    Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292,

    Pith/arXiv arXiv

  4. [7]

    P., Surana, H., Agarwal, D., Mishra, B

    Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., V ora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models.arXiv preprint arXiv:2407.01725,

    Pith/arXiv arXiv

  5. [8]

    C., Barabasi, D

    Mitchener, L., Yiu, A., Chang, B., Bourdenx, M., Nadol- ski, T., Sulovari, A., Landsness, E. C., Barabasi, D. L., Narayanan, S., Evans, N., et al. Kosmos: An ai scientist for autonomous discovery.arXiv preprint arXiv:2511.02824,

    Pith/arXiv arXiv

  6. [9]

    Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

    Si, C., Yang, D., and Hashimoto, T. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109,

    Pith/arXiv arXiv

  7. [11]

    The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API

    8 Adversarial Fast-Moving Real-World Domains for Evaluating AI Scientists Supplementary Methods Models Six frontier LLMs spanning three providers were evaluated. The OpenAI models GPT-5.2 (August 2025 knowledge cutoff) and o3 (June 2024 knowledge cutoff) were accessed via the OpenAI API. The Google models Gemini 3 Flash and Gemini 3.1 Pro (both January 20...

    work page 2025

  8. [12]

    F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel

    and Lorwyn Eclipsed (released 23 January 2026), so neither corpus could have appeared in any model’s training data. F1 generated ideas and real innovations were embedded with OpenAI’s text-embedding-3-smallmodel. Ground-truth collection The reference set of 40 real 2026 pre-season innovations was compiled from publicly available technical analysis publish...

    work page 2026

  9. [15]

    The Top 15 by Day 2 standings and four featured archetype-diverse builds were collected from official PT Lorwyn Eclipsed coverage (30 January – 1 February 2026)

    15 magic.gg/events/pro-tour-lorwyn- eclipsed PT Lorwyn Eclipsed (featured archetype- diverse builds) 4 magic.gg/events/pro-tour-lorwyn- eclipsed Total 19 Table 3.Provenance of the 19 PT Lorwyn Eclipsed reference decks. The Top 15 by Day 2 standings and four featured archetype-diverse builds were collected from official PT Lorwyn Eclipsed coverage (30 Janu...

    work page 2026

  10. [2015]

    S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al

    Starace, G., Jaffe, O., Sherburn, D., Aung, J., Chan, J. S., Maksin, L., Dias, R., Mays, E., Kinsella, B., Thompson, W., et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848,

    Pith/arXiv arXiv

  11. [2021]

    Hypobench: Towards systematic and principled benchmarking for hy- pothesis generation.arXiv preprint arXiv:2504.11524,

    7 Adversarial Fast-Moving Real-World Domains for Evaluating AI Scientists Liu, H., Huang, S., Hu, J., Zhou, Y ., and Tan, C. Hypobench: Towards systematic and principled benchmarking for hy- pothesis generation.arXiv preprint arXiv:2504.11524,

    arXiv

  12. [2024]

    The need for verification in ai-driven scientific discov- ery.arXiv preprint arXiv:2509.01398,

    Cornelio, C., Ito, T., Cory-Wright, R., Dash, S., and Horesh, L. The need for verification in ai-driven scientific discov- ery.arXiv preprint arXiv:2509.01398,

    arXiv

  13. [2025]

    com/DGoettlich/history-llms

    URL https://github. com/DGoettlich/history-llms. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Palepu, A., Sirkovic, P., Myaskovsky, A., Weissenberger, F., Rong, K., Tanno, R., et al. Towards an ai co-scientist.arXiv preprint arXiv:2502.18864,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.