Pith. sign in

REVIEW 3 major objections 7 minor 53 references

Best AI scientist hits 27% accuracy tracing scientific idea lineage

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 01:40 UTC pith:6N2GNIRY

load-bearing objection New benchmark for scientific lineage reasoning; the 'compositional bottleneck' claim is underdetermined without per-field accuracy data. the 3 major comments →

arxiv 2607.08758 v1 pith:6N2GNIRY submitted 2026-07-09 cs.AI

Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation

Yifan Zhou , Qihao Yang , Yan Li , Donggang Li , Xiru Hu , Hokin Deng , Ziyang Gong , Xuanyi Zhou
show 9 more authors
Huacan Wang Xiangchao Yan Wanghan Xu Wenlong Zhang Shaofeng Zhang Yue Zhou Yifan Yang Zhihang Zhong Xue Yang
This is my paper
classification cs.AI
keywords lineageideareasoninggenomeobjectsscientificgenerationinheritance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that scientific ideas carry an inheritance structure — mechanisms are inherited, limitations are repaired, components are recombined — and that this structure can be made explicit and auditable. The authors define an Idea Genome as a minimal, typed, evidence-grounded idea object extracted from a paper (with roles like niche, mechanism, limitation, delta, claim), and a GenomeDiff as the alignment of these objects across a predecessor and successor, recording which ideas are inherited, mutated, lost, imported, or newly inserted. They classify transitions into six evolutionary dynamics: mutation, adaptive radiation, hybridization, speciation, niche competition, and isolation. The benchmark built on this framework, IG-Bench, tests whether AI systems can abstract genomes from papers, trace inheritance across multiple papers, reason about evolutionary transitions, verify lineage claims, and generate proposals that coherently descend from a given lineage. The central empirical finding is a compositional bottleneck: the strongest system reaches only 27.3% exact accuracy on closed-form lineage reasoning, because models recover local signals but fail to keep parent identity, driver mechanism, object fate, and verification flags jointly consistent. In generation, structured lineage context does not uniformly help — it reshuffles system rankings, separating systems that can operationalize lineage evidence from those that merely benefit from more text.

Core claim

The paper discovers that the gap between plausible research text and genuine lineage competence is measurable and large. Even frontier LLMs that write fluent proposals fail when asked to identify which specific mechanism a new idea inherits, which limitation it repairs, and whether a claimed lineage is coherent rather than merely topical. The failure is compositional: individual components (identifying a parent paper, inferring a driver mechanism, classifying an object's fate) are each partially solvable, but holding all of them consistent simultaneously is where systems break down. In generation tasks, the Population-Evolution Score reveals that proposals can score high on variation (soudue

What carries the argument

Idea Genome — a minimal, typed, evidence-grounded idea object extracted from a paper, with role types (niche, mechanism, observation, limitation, delta, claim). GenomeDiff — the alignment of Idea Genome objects across a predecessor and successor, recording inheritance, mutation, loss, external import, or novel insertion, plus a primary transition driver. Six evolutionary dynamics classify GenomeDiff patterns: mutation (driver mechanism inherited with local changes), adaptive radiation (driver persists but moves to a new task or domain), hybridization (driver objects imported from two or more distinct lineages), speciation (driver mechanism replaced by a new lineage-forming mechanism), niche

Load-bearing premise

The entire framework depends on the claim that typed Idea Genome objects and six evolutionary dynamics are a non-arbitrary, sufficient representation of scientific inheritance — if these categories are artifacts of the authors' annotation choices rather than natural joints in the structure of ideas, the benchmark measures consistency with a particular labeling scheme rather than genuine lineage competence.

What would settle it

If an AI system could achieve high exact-match accuracy on IG-Exam by exploiting surface textual patterns rather than genuinely reasoning about inheritance structure, the compositional bottleneck would be an artifact of the benchmark's format rather than a real capability gap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the Idea Genome representation captures real scientific inheritance structure, auto-research systems need compositional verification modules — not just better retrieval — to generate ideas that genuinely descend from prior work.
  • The finding that lineage context reshuffles rankings rather than uniformly helping suggests that evaluation of AI scientists should test whether systems can use structured evidence, not just whether they produce more text when given more context.
  • The moderate correlation between closed-form lineage understanding and generation heredity implies that improving reasoning benchmarks may transfer to generation quality, but the two remain complementary skills.
  • The dynamics classification (mutation vs. speciation vs. hybridization) provides a vocabulary for auditing whether a generated idea is a genuine descendant or merely co-located with its claimed parent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the compositional bottleneck generalizes beyond this benchmark, it suggests that current scaling approaches (more parameters, more retrieval) will not close the lineage-reasoning gap without architectural changes that enforce multi-constraint consistency.
  • The distinction between variation and heredity in the PES metric could be applied to human peer review: reviewers often reward novelty without checking whether a paper actually inherits and repairs the right limitation, which is the same blind spot the benchmark exposes in AI.
  • The six evolutionary dynamics could serve as a diagnostic taxonomy for detecting weak or fraudulent lineage claims: a paper claiming descent from lineage X but showing no driver inheritance would be classified as niche competition rather than genuine lineage.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces IG-Bench, a benchmark for evaluating whether AI systems can reason about scientific lineage—the inheritance of mechanisms, limitations, and ideas across papers. The core representational contribution is the IdeaGene framework, which decomposes papers into typed Idea Genome objects (niche, mechanism, observation, limitation, delta, claim) and aligns them via GenomeDiff records under six evolutionary dynamics. The benchmark has two parts: IG-Exam (42 task types, 1,029 instances) for closed-form lineage reasoning, and IG-Arena for lineage-grounded idea generation scored by a Population-Evolution Score (PES). Experiments on 14 LLM-based scientists show a best exact accuracy of 27.3% on IG-Exam and heterogeneous gains from structured lineage context in IG-Arena. The paper is well-motivated, the framework is clearly formalized, and the empirical findings are substantive.

Significance. The paper addresses a genuine gap: existing benchmarks evaluate retrieval, factuality, or generation fluency, but not whether systems can trace mechanism-level inheritance across papers. The IdeaGene framework provides a falsifiable, auditable representation layer (typed objects with evidence pointers, explicit alignment rules), which is a meaningful step beyond paper-level citation analysis. The finding that tool scaffolds help retrieval (T2) but not compositional verification (T4) is actionable for the auto-research community. The benchmark covers 10 domains and includes human validation (84.7% inter-annotator agreement on dynamics labels, 80% human-judge concordance). These are concrete strengths.

major comments (3)
  1. §5.2 and §5.4, Finding 1: The central interpretive claim is that LLM-based scientists face a 'compositional bottleneck' in lineage reasoning. This claim rests on exact-match accuracy (§4.2: 'every required field must be correct at the same time'), with the headline figure of 27.3%. However, the paper never reports per-field accuracy rates. Without these, one cannot distinguish a genuine compositional reasoning failure (where models specifically fail to integrate information across fields) from a statistical artifact of exact-match scoring, where independent per-field errors compound multiplicatively. If per-field accuracy is approximately 60-70%, a joint accuracy of ~27% could be roughly what independent errors would produce, requiring no compositional bottleneck. The T1→T4 gradient (34.4%→37.9%→25.3%→17.4%) is suggestive but confounded by task-type differences. The paper should report (
  2. §4.3 and §5.3: The PES metric relies on a judge panel of 3 model judges (§4.3), while the benchmark's golden labels are constructed via 'LLM-assisted extraction followed by expert audit' (§4.1). This creates a potential circularity: the same class of models being evaluated is also involved in generating the evaluation substrate and scoring the outputs. The paper reports human concordance of 80% with the model-judge panel and Krippendorff's α=0.74, which partially addresses reliability. However, the paper does not report which model(s) were used for extraction vs. judging, nor whether the judge models overlap with the evaluated systems. A clarification of the separation between extraction models, judge models, and evaluated systems would address this concern. Additionally, reporting PES scores from a human-judge subset (even small) on the same proposals would strengthen the metric's claim
  3. §3.5 and Table 2: The six evolutionary dynamics (mutation, adaptive radiation, hybridization, speciation, niche competition, isolation) are presented as operational categories, but the priority rules for ambiguous cases ('Hybridization before Speciation, Speciation before Niche Competition when lineage evidence exists, and Adaptive Radiation before Mutation when the setting shift is the driver') appear arbitrary. The paper does not report how frequently these priority rules are invoked during annotation, nor how sensitive the dynamics labels are to the choice of priority ordering. If a substantial fraction of labels depend on these tie-breaking rules, the dynamics classification becomes less informative. Reporting the percentage of GenomeDiff records where the priority rule was invoked, and a sensitivity analysis on the ordering, would address this.
minor comments (7)
  1. §4.1: The paper states that construction involves '50 graduate annotators' but does not report how many annotators worked on each domain, or the per-domain agreement rates. Given that the benchmark spans 10 domains, reporting domain-level agreement would strengthen the quality assurance claim.
  2. Table 4: The system names reference future model versions (e.g., 'GPT-5.5', 'Claude Opus 4.7', 'Gemini-3.1-pro-preview', 'DeepSeek-V4-Pro', 'GLM-5.1', 'MiniMax-M2.7', 'Qwen3.6-Max-Preview'). The paper should clarify whether these are actual production models or internal previews, and include version dates, as this affects reproducibility.
  3. Figure 3a: The PES gains from Question to Lineage are labeled with individual delta values, but the y-axis range (65-90) compresses the visual differences. A wider y-axis or a dedicated delta plot would make the heterogeneity of gains more visually evident.
  4. §4.1, Quality Assurance paragraph: The inter-annotator agreement of 84.7% is reported 'before adjudication,' but the post-adjudication agreement is not reported. Reporting both would give a clearer picture of the dynamics labels.
  5. Appendix B.1, T2-04 example: The lineage grouping example partitions 8 objects into 2 groups, but the rationale for why exactly 2 groups (rather than 3 or 4) is the correct answer is not fully justified. A brief note on the annotation standard for group count would help.
  6. §5.3: The Spearman ρ=0.82 between PES and ELO is reported as 'partial divergence,' but this is actually a fairly strong correlation. The paper should clarify what magnitude of divergence would be concerning, or frame this more carefully.
  7. The paper uses the evolutionary metaphor extensively (genomes, mutation, speciation, etc.). While the authors acknowledge this is 'operational' rather than biological (§3), the metaphor occasionally obscures the mechanics. For instance, 'selection value for future research' (§4.3) is a metaphor for downstream impact, but the term 'selection' may confuse readers expecting a population genetics mechanism. A brief glossary or more neutral terminology in key definitions would help.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's construction and evaluation are self-contained against external human-validated ground truth, with no derivation chain that reduces to its own inputs.

full rationale

This is a benchmark paper, not a derivation paper, so the circularity patterns (self-definitional, fitted-input-called-prediction, self-citation load-bearing, uniqueness imported, ansatz smuggled, renaming known result) do not naturally apply. The two potential circularity concerns — (1) LLM-assisted extraction for ground truth and (2) model-judge panels for IG-Arena scoring — are both mitigated by independent human validation: 50 graduate annotators validated GenomeDiff labels (84.7% inter-annotator agreement before adjudication), stratified human solving confirmed IG-Exam difficulty reflects compositional challenge rather than label noise, and human judges reached 80% agreement with the model-judge panel on IG-Arena pairwise rankings (§4.1, Appendix D). The PES metric (Eq. 3) is an arithmetic mean of three independently-scored dimensions (Heredity, Variation, Selection), not a quantity defined in terms of its own inputs. The 'compositional bottleneck' claim is an empirical finding from exact-match scoring, not a derived result. The skeptic's concern about per-field accuracy compounding is a correctness/interpretation risk, not a circularity: the 27.3% figure is measured against externally-audited gold labels, not predicted from a fitted parameter or defined by construction. No step in the paper's chain reduces to its inputs by definition or self-citation.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 2 invented entities

The axiom ledger reveals that the paper's contribution rests on several design choices (role types, dynamics categories) and methodological axioms (LLM-assisted golden labels, model-judge panels). These are not flaws in themselves, but they represent the framework's foundational assumptions.

free parameters (2)
  • Six evolutionary dynamics categories
    The choice of six specific dynamics (mutation, adaptive radiation, hybridization, speciation, niche competition, isolation) is a design choice that structures the entire benchmark. The priority rule for ambiguous cases (Hybridization before Speciation, etc.) is also a manually defined parameter.
  • Idea Genome role types = niche, mechanism, observation, limitation, delta, claim
    The six role types are the fundamental unit of analysis. Their selection is an ad-hoc design decision to make lineage comparison tractable, not a derived necessity.
axioms (3)
  • domain assumption Scientific ideas can be decomposed into minimal, typed, evidence-grounded objects that are inherited, mutated, or lost across papers.
    This is the foundational assumption of the IdeaGene framework (§3.2). The entire benchmark assumes this decomposition is meaningful and captures 'lineage competence.'
  • ad hoc to paper LLM-assisted extraction followed by expert audit produces 'golden' lineage traces.
    The benchmark construction (§4.1) relies on this pipeline to generate ground truth. The validity of the benchmark depends on this axiom holding.
  • ad hoc to paper Model-judge panels can reliably score lineage-grounded generation quality.
    The IG-Arena PES metric (§4.3) relies on 3 model judges. The paper validates this with 80% human concordance, but the use of model judges as the primary metric is an axiom of the evaluation methodology.
invented entities (2)
  • Idea Genome no independent evidence
    purpose: A minimal, typed, evidence-grounded idea object extracted from a paper for lineage comparison.
    This is the core invented entity of the framework. Its 'independent evidence' is the benchmark itself and the human agreement rates, but it has no falsifiable handle outside the paper's own annotation scheme.
  • GenomeDiff no independent evidence
    purpose: An alignment of Idea Genome objects across predecessor and successor work, recording inheritance, mutation, loss, etc.
    The diff record is the operational unit for measuring lineage. Like the Idea Genome, it is an invented entity for evaluation purposes.

pith-pipeline@v1.1.0-glm · 23345 in / 2130 out tokens · 284731 ms · 2026-07-10T01:40:53.856685+00:00 · methodology

0 comments
read the original abstract

Scientific ideas rarely start from a blank page. They inherit mechanisms, repair known limitations, and recombine pieces of earlier work, much like biological genomes. Current benchmarks still say little about whether AI systems can follow this inheritance structure. We present IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. IG-Bench is organized around the IdeaGene framework: each paper or proposal is represented as a set of minimal, typed, evidence-grounded Idea Genome objects, and a GenomeDiff aligns these objects to record inheritance, mutation, loss, external import, and novel insertion under six operational evolutionary dynamics. The benchmark contains 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records across 10 scientific domains. It supports two evaluations. IG-Exam (42 task types, 1,029 instances) tests closed-form lineage reasoning across Idea Genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification. IG-Arena evaluates generation with a lineage-conditioned Population-Evolution Score(PES), asking whether a proposal can be inserted as a coherent descendant of a given lineage population: it should inherit the right Idea Genome objects, vary meaningfully from nearby work, and offer selection value for future research. Experiments on 14 LLM-based scientists expose a compositional bottleneck. The strongest system reaches only 27.3% exact accuracy on lineage reasoning, and structured lineage context reshuffles system rankings rather than helping every participant uniformly.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 53 canonical work pages · 18 internal anchors

  1. [1]

    LitSearch: A retrieval benchmark for scientific literature search

    Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. LitSearch: A retrieval benchmark for scientific literature search. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15068–15083, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.e...

    work page doi:10.18653/v1/2024.emnlp-main.840 2024

  2. [2]

    Synthesizing scientific literature with retrieval-augmented language models

    Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, et al. Synthesizing scientific literature with retrieval-augmented language models. Nature, 650:857–863, 2026. doi: 10.1038/s41586-025-10072-4

    work page doi:10.1038/s41586-025-10072-4 2026

  3. [3]

    ResearchAgent: Iterative research idea generation over scientific literature with large language models

    Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. ResearchAgent: Iterative research idea generation over scientific literature with large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pag...

    work page doi:10.18653/v1/2025.naacl-long.342 2025

  4. [4]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. doi: 10.1093/biomet/39.3-4.324

    work page doi:10.1093/biomet/39.3-4.324 1952

  5. [5]

    End-to-end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. InProceedings of ECCV, 2020. doi: 10.1007/978-3-030-58452-8_13

    work page doi:10.1007/978-3-030-58452-8_13 2020

  6. [6]

    Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference.arXiv preprint arXiv:2403.04132, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  7. [7]

    Structural scaffolds for citation intent classification in scientific publications

    Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. Structural scaffolds for citation intent classification in scientific publications. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3586–3596, 2019. doi: 10.18653/v1/N19-1361

    work page doi:10.18653/v1/n19-1361 2019

  8. [8]

    John Murray, 1859

    Charles Darwin.On the Origin of Species by Means of Natural Selection. John Murray, 1859

    work page

  9. [9]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of NAACL-HLT, 2019. doi: 10.18653/v1/N19-1423

    work page doi:10.18653/v1/n19-1423 2019

  10. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale.Proceedings of ICLR, 2021

    Alexey Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale.Proceedings of ICLR, 2021

    work page 2021

  11. [11]

    Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators.arXiv preprint arXiv:2404.04475, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Elo.The Rating of Chessplayers, Past and Present

    Arpad E. Elo.The Rating of Chessplayers, Past and Present. Arco, 1978

    work page 1978

  13. [13]

    Bergstrom, Katy Börner, James A

    Santo Fortunato, Carl T. Bergstrom, Katy Börner, James A. Evans, Dirk Helbing, Staša Milojević, Alexander M. Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, Alessandro Vespignani, Ludo Waltman, Dashun Wang, and Albert-László Barabási. Science of science.Science, 359(6379):eaao0185, 2018. doi: 10.1126/science.aao0185

    work page doi:10.1126/science.aao0185 2018

  14. [14]

    Alireza Ghafarollahi and Markus J. Buehler. SciAgents: Automating scientific discovery through bioinspired multi-agent intelligent graph reasoning.Advanced Materials, 37(22):2413523, 2025. doi: 10.1002/adma.202413523

    work page doi:10.1002/adma.202413523 2025

  15. [15]

    Nova: An Iterative Planning and Search Approach to Enhance Novelty and Diversity of LLM Generated Ideas

    Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu Lu, Yaochu Jin, Lili Pan, and Zhenzhong Lan. Nova: An iterative planning and search approach to enhance novelty and diversity of LLM generated ideas.arXiv preprint arXiv:2410.14255, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    SWE-bench: Can language models resolve real-world github issues? InProceedings of ICLR, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? InProceedings of ICLR, 2024

    work page 2024

  17. [17]

    Measuring the evolution of a scientific field through citation frames.Transactions of the Association for Computational Linguistics, 6:391–406, 2018

    David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. Measuring the evolution of a scientific field through citation frames.Transactions of the Association for Computational Linguistics, 6:391–406, 2018

    work page 2018

  18. [18]

    Computing krippendorff’s alpha-reliability

    Klaus Krippendorff. Computing krippendorff’s alpha-reliability. Departmental Paper 43, Annenberg School for Communication, University of Pennsylvania, 2011. URLhttps://repository.upenn.edu/asc_papers/43

    work page 2011

  19. [19]

    Kuhn.The Structure of Scientific Revolutions

    Thomas S. Kuhn.The Structure of Scientific Revolutions. University of Chicago Press, 1962. 12

    work page 1962

  20. [20]

    Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents

    Long Li, Weiwen Xu, Jiayan Guo, Ruochen Zhao, Xingxuan Li, Yuqian Yuan, Boqiang Zhang, Yuming Jiang, Yifei Xin, Ronghao Dang, Deli Zhao, Yu Rong, Tian Feng, and Lidong Bing. Chain of ideas: Revolutionizing research via novel idea development with LLM agents.arXiv preprint arXiv:2410.13185, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  21. [21]

    Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    work page 2023

  22. [22]

    WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking LLMs with challenging tasks from real users in the wild.arXiv preprint arXiv:2406.04770, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InAdvances in Neural Information Processing Systems, volume 36, pages 34892–34916, 2023

    work page 2023

  24. [24]

    AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration

    Jiaqi Liu, Shi Qiu, Mairui Li, Bingzhou Li, Haonian Ji, Siwei Han, Xinyu Ye, Peng Xia, Zihan Dong, Meng Chen, Congyu Zhang, Letian Zhang, Guiming Chen, Haoqin Tu, Xinyu Yang, Lu Feng, Xujiang Zhao, Haifeng Chen, Jiawei Zhou, Xiao Wang, Weitong Zhang, Hongtu Zhu, Yun Li, Jieru Mei, Hongliang Fei, Jiaheng Zhang, Linjie Li, Linjun Zhang, Yuyin Zhou, Sheng Wa...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Multi-xscience: A large-scale dataset for extreme multi-document summarization of scientific articles

    Yao Lu, Yue Dong, and Laurent Charlin. Multi-xscience: A large-scale dataset for extreme multi-document summarization of scientific articles. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8068–8074, 2020. doi: 10.18653/v1/2020.emnlp-main.648

    work page doi:10.18653/v1/2020.emnlp-main.648 2020

  27. [27]

    GAIA: a benchmark for General AI Assistants

    Grégoire Mialon et al. GAIA: A benchmark for general ai assistants.arXiv preprint arXiv:2311.12983, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  28. [28]

    Language mod- els are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language mod- els are unsupervised multitask learners. Technical report, OpenAI, 2019. URL https://cdn.openai.com/ better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    work page 2019

  29. [29]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, pages 8748–8763, 2021

    work page 2021

  30. [30]

    YOLO9000: Better, faster, stronger

    Joseph Redmon and Ali Farhadi. YOLO9000: Better, faster, stronger. InProceedings of CVPR, 2017

    work page 2017

  31. [31]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. InProceedings of CVPR, 2016. doi: 10.1109/CVPR.2016.91

    work page doi:10.1109/cvpr.2016.91 2016

  32. [32]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of EMNLP-IJCNLP, 2019. doi: 10.18653/v1/D19-1410

    work page doi:10.18653/v1/d19-1410 2019

  33. [33]

    Faster R-CNN: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. InAdvances in Neural Information Processing Systems, 2015

    work page 2015

  34. [34]

    ARB: Advanced Reasoning Benchmark for Large Language Models

    Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J. Nay, Kshitij Gupta, and Aran Komatsuzaki. ARB: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  35. [35]

    Agent Laboratory: Using LLM Agents as Research Assistants

    Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using LLM agents as research assistants.arXiv preprint arXiv:2501.04227, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Assisting in writing wikipedia-like articles from scratch with large language models

    Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, and Monica Lam. Assisting in writing wikipedia-like articles from scratch with large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6252–6278, ...

    work page doi:10.18653/v1/2024.naacl-long.347 2024

  37. [37]

    Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

    Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers.arXiv preprint arXiv:2409.04109, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  38. [38]

    SciRepEval: A multi-format benchmark for scientific document representations

    Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. SciRepEval: A multi-format benchmark for scientific document representations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.338

    work page doi:10.18653/v1/2023.emnlp-main.338 2023

  39. [39]

    SciEval: A multi-level large language model evaluation benchmark for scientific research.Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19053–19061, 2024

    Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu. SciEval: A multi-level large language model evaluation benchmark for scientific research.Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19053–19061, 2024. doi: 10.1609/aaai.v38i17.29872

    work page doi:10.1609/aaai.v38i17.29872 2024

  40. [40]

    AI-Researcher: Autonomous Scientific Innovation

    Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous scientific innovation. arXiv preprint arXiv:2505.18705, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  41. [41]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems, 2017

    work page 2017

  42. [42]

    Text Embeddings by Weakly-Supervised Contrastive Pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training.arXiv preprint arXiv:2212.03533, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  43. [43]

    Large Language Models are not Fair Evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators.arXiv preprint arXiv:2305.17926, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  44. [44]

    SciMON: Scientific inspiration machines optimized for novelty

    Qingyun Wang, Doug Downey, Heng Ji, and Tom Hope. SciMON: Scientific inspiration machines optimized for novelty. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 279–299, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.18

    work page doi:10.18653/v1/2024.acl-long.18 2024

  45. [45]

    SciEvalKit: An open-source evaluation toolkit for scientific general intelligence.arXiv preprint arXiv:2512.22334, 2025

    Yiheng Wang et al. SciEvalKit: An open-source evaluation toolkit for scientific general intelligence.arXiv preprint arXiv:2512.22334, 2025. doi: 10.48550/arXiv.2512.22334

    work page doi:10.48550/arxiv.2512.22334 2025

  46. [46]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. InProceedings of the Third Workshop on Noisy User-generated Text, 2017. doi: 10.18653/v1/W17-4413

    work page doi:10.18653/v1/w17-4413 2017

  47. [47]

    Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists

    Yujun Wu, Dongxu Zhang, Xinchen Li, Jinhang Xu, Yiling Duan, Yumou Liu, Jiabao Pan, Qiyuan Zhu, Xuanhe Zhou, Jingxuan Wei, Siyuan Li, Jintao Chen, Conghui He, and Cheng Tan. Intern-Atlas: A methodological evolution graph as research infrastructure for AI scientists.arXiv preprint arXiv:2604.28158, 2026. doi: 10.48550/ arXiv.2604.28158

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  48. [48]

    Probing scientific general intelligence of LLMs with scientist-aligned workflows.arXiv preprint arXiv:2512.16969, 2025

    Wanghan Xu et al. Probing scientific general intelligence of LLMs with scientist-aligned workflows.arXiv preprint arXiv:2512.16969, 2025. doi: 10.48550/arXiv.2512.16969

    work page doi:10.48550/arxiv.2512.16969 2025

  49. [49]

    ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

    Wanghan Xu et al. ResearchClawBench: A benchmark for end-to-end autonomous scientific research.arXiv preprint arXiv:2606.07591, 2026. doi: 10.48550/arXiv.2606.07591

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.07591 2026

  50. [50]

    The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Shunyu Yao et al.τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  52. [52]

    Deep ideation: Designing LLM agents to generate novel research ideas on scientific concept network.arXiv preprint arXiv:2511.02238, 2025

    Keyu Zhao, Weiquan Lin, Qirui Zheng, Fengli Xu, and Yong Li. Deep ideation: Designing LLM agents to generate novel research ideas on scientific concept network.arXiv preprint arXiv:2511.02238, 2025

    work page arXiv 2025

  53. [53]

    Provides a dedicated probabilistic method for unbiased learning of antibody indel statistics, compared with prior work relying on biased annotation-based indel calls

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, et al. Judging LLM-as-a-judge with MT-Bench and chatbot arena.Advances in Neural Information Processing Systems, 36, 2023. 14 Appendix A Main-Challenge Task Inventory Table 5Main IG-Exam challenge: 42 task types across 4 capa...

    work page 2023