Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

Automated Capability Discovery via Foundation Model Self-Exploration

T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A foundation model assigned as scientist can automatically probe another model's capabilities and failure modes without human task design, and human surveys largely confirm its scoring.

desk verdict A useful automated task-generation pipeline for capability mapping, but the 'surprising' part is manually curated and the abstract oversells it. read the letter →

arxiv 2502.07577 v3 pith:T3ULIXID submitted 2025-02-11 cs.LG cs.AIcs.CL

Cong Lu , Shengran Hu , Jeff Clune This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords AutomatedCapabilityDiscoveryfoundationmodelevaluationopen-endednessLLM-as-a-judgetaskfamilygenerationreportAIsafetyself-exploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Automated Capability Discovery (ACD), a loop in which one foundation model acts as a scientist that writes new tasks, tests a subject model on them, scores the results, and keeps only tasks deemed interestingly new. The claim is that this process automatically maps a model's strengths and weaknesses into a broad landscape of capability areas and failure modes, the way a human evaluator would by trial and error, but at far lower cost. On GPT-4o acting as both scientist and subject, ACD generated 1,330 novel tasks over 5,000 generations that clustered into 25 capability areas, including surprising failures on simple arithmetic sequences and successes on hard logic puzzles. Human surveys found 92.2% of the generated tasks clear and valid and an F1 of 0.86 between the automated judge and human correctness judgments, supporting the claim that model-generated evaluation can stand in for human evaluation at scale.

What carries the argument

The load-bearing mechanism is the iterative ACD loop built on the METR Task Standard. A scientist model maintains an archive of task families; at each iteration it sees a sample of prior tasks, proposes a new family written as a Python class with get_tasks, get_instructions, and score, reflects on errors, and asks whether the proposal is 'interestingly new' relative to embedding neighbors using text-embedding-3-small. The subject model then attempts each task with chain-of-thought; scoring is programmatic where possible and otherwise delegated to a GPT-4o-based judge. Finally, tasks are embedded, visualized by t-SNE, and clustered by HDBSCAN; consistent success or failure within a cluster is read as a capability or failure mode, and a scientist-generated natural language report summarizes the clusters.

What would settle it

Take the archived tasks that humans rated 'Very Difficult' and ask a second human panel to independently score every subject response; if judge-human agreement on that subset is substantially below the overall F1 of 0.86, the claim that automated scoring validates capability discovery on hard tasks fails. A cheaper check: run ACD with the same scientist and subject but with a different judge model (e.g., Claude instead of GPT-4o) and compare the resulting capability reports; large divergences in which failures are flagged would show the discovery is partly an artifact of the judge.

Watch

Extended reading notes

Core claim

ACD's central discovery is that a foundation model, prompted with an archive of prior tasks, can propose open-ended Python-coded task families that reveal a subject model's capabilities, and that the subject's own generation or a separate judge model can grade them reliably enough to build a map. The authors show the loop converges to a stable 'capability signature': repeated runs across seeds produce similar task distributions, and the same archive re-administered to a weaker model (Llama3-8B) yields a performance profile that is nearly a subset of GPT-4o's, with narrower gaps on creative writing. They also show the scientist model matters: with Claude Sonnet 3.5 as scientist, the discovered tasks shift toward interdisciplinary and creative domains, indicating that the probe is shaped by whoever designs it. The paper treats a 'capability' or 'failure mode' as a consistent performance pattern across a family of automatically generated tasks, not as a fixed property, and argues that this operational definition is what makes automated discovery tractable.

Load-bearing premise

The paper treats the automated judge's binary success/failure labels as ground truth for defining every discovered capability and failure mode, so if the judge is wrong on the hardest tasks, the corresponding capability map is wrong in exactly the places where surprises matter most.

Editorial extensions

If this is right

  • ACD produces an automatically compiled Capability Report that lets developers scan a model's strengths and failure modes before deployment, reducing the manual effort of benchmark design.
  • A single GPT-4o-discovered task archive can be re-administered to future or weaker models under identical instructions and scoring, giving direct cross-model comparisons without new human task writing.
  • Because the loop still finds roughly 20% of proposed tasks interestingly new after 5,000 generations, the discovered map is not saturated; running longer should reveal more capabilities.
  • Using different scientist models (e.g., Claude Sonnet 3.5 vs GPT-4o) broadens coverage of the same subject model, motivating an ensemble of scientists for fuller capability maps.
  • The automated judge's high agreement with humans (F1 0.86) supports scaling evaluation: thousands of tasks can be scored for about $0.10 per task, a cost low enough for routine pre-deployment screening.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If judge accuracy is lower on the very hardest discovered tasks, as Figure 3(c) suggests, then the failure-mode portions of capability reports on hard task clusters are the least trustworthy; a human audit focused on low-scoring, high-difficulty clusters would be a targeted extension of this work.
  • When scientist, judge, and subject are the same model, discovered capabilities may partly reflect the judge's own blind spots rather than the subject's; comparing judge models or adding a second judge would test this boundary.
  • The 'interestingly new' filter depends on an embedding model and a fixed GPT-4o novelty checker, so the diversity frontier is partially defined by that checker; swapping the novelty model should shift what counts as a capability area.
  • The open-ended loop could be redirected from evaluation to training: tasks the subject consistently fails are natural curriculum material, though the paper only gestures at self-improvement as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces Automated Capability Discovery (ACD), a framework in which one foundation model (the scientist) proposes open-ended tasks, filters them for novelty, evaluates a subject model (potentially itself) using programmatic checks or an LLM judge, clusters the resulting tasks, and generates a capability report. The authors demonstrate ACD on GPT-4o, Claude Sonnet 3.5, and Llama3-8B, reporting thousands of discovered tasks across dozens of clusters. A human survey on GPT-4o-discovered tasks finds 92.2% of tasks clear and valid and an F1 of 0.86 between automated and human scoring. The paper argues that ACD automatically and systematically uncovers surprising capabilities and failures, and it open-sources code and logs.

Significance. If the central claims hold, ACD is a valuable step toward scalable, low-cost automated evaluation of foundation models, potentially complementing human benchmark design and helping identify unknown unknowns. The paper's strengths include the open-sourcing of code and evaluation logs, the large-scale human validation of task validity and scoring, and the breadth of experiments across multiple scientist-subject pairings. The human survey provides a significant independent anchor for task quality, although it does not fully validate the 'surprisingness' selection or judge accuracy on the hardest tasks. The paper also acknowledges several limitations and future work, which is commendable.

major comments (3)
  1. [Abstract and Section E.2] The abstract's central claim that ACD 'automatically and systematically uncovers a diverse spectrum of surprising capabilities and failures' is not supported by the system as described. Section E.2 (Manual Selection of Surprising Tasks) states that both crowd-based and model-based filters for surprisingness failed and that the authors 'ultimately opted for a manual curation step led by the authors.' The headline examples in Figure 1 and Section E.3 are therefore hand-picked, not outputs of the automated pipeline. The automated Capability Report does include an example-selection step (Section H.2.2), but its selections are not validated against human surprise judgments. Please either revise the central claims to distinguish automated task discovery from manually curated illustrative examples, or add and validate an automated surprisingness filter.
  2. [Section 5.1 and Appendix F] The automated judge's success/failure labels are used to compute cluster success rates and to label capabilities and failure modes, but its accuracy on the hardest discovered tasks is not established. Figure 3(c) shows that judge-human F1 drops on 'Very Difficult' tasks, and Appendix F provides concrete cases where the judge marks correct responses as failures (e.g., the frustum volume and the algebraic equation). Since 'surprising' failures and capabilities are identified relative to these scores, the characterization of the most difficult task families may be unreliable. Please report judge accuracy separately by difficulty and either strengthen the judge or qualify capability claims for tasks where judge accuracy is unverified.
  3. [Section 5.2 and Tables 4-6] Claims such as 'Llama3-8B struggles substantially more than GPT-4o' are based on cluster-level success rates without uncertainty intervals, despite large variation in cluster sizes (from 16 to 382 tasks). For example, clusters 21 and 22 in Table 4 have success rates of 54.8% and 84.0% with only 31 and 25 tasks, respectively; these differences may not be meaningful. Provide confidence intervals or otherwise quantify uncertainty in the cluster success rates that underpin the cross-model comparison.
minor comments (3)
  1. [Figure 4] The radial axis labels are too small to read, as the caption itself acknowledges; a larger figure or a table of cluster success rates would improve readability.
  2. [Section 5.3] The attribution of task-style differences to Claude Sonnet 3.5's 'creative personality' is speculative and not supported by controlled analysis; consider softening this wording.
  3. [Section C.1] The cost estimate of '$0.1 per task created' is not fully specified; clarify whether it includes all API calls (scientist proposals, embeddings, judge evaluations, and subject evaluations) and how variance across settings is handled.

Circularity Check

1 steps flagged · score 5.0 of 10

The automated pipeline is mostly self-contained, but the 'surprising' headline examples are manually curated (E.2), so part of the central claim reduces to author selection rather than automatic discovery.

  1. fitted input called prediction [Section E.2 ('Manual Selection of Surprising Tasks'), with the claim in the Abstract and Section 5.4]
    "We initially experimented with two main more scalable approaches to filter for surprising tasks: 1. Crowd-based filtration... 2. Model-based filtration... Given these challenges, we ultimately opted for a manual curation step led by the authors. While this approach is more labor-intensive and subjective, it allowed us to present a set of especially illustrative or counterintuitive results..."

    The abstract states that ACD 'automatically and systematically uncovers a diverse spectrum of surprising capabilities and failures', but E.2 concedes that the automated surprise filters failed and that the authors manually chose the examples. Consequently the surprising successes and failures showcased in Figure 1 and Section E.3 are selected by the authors from the archive, not produced by ACD's automatic pipeline. The automatic example-selection prompt in H.2.2 is not validated against human surprise judgments, so the 'automatically... surprising' component of the headline claim is supplied by human editorial input rather than derived by the algorithm.

full rationale

The core ACD loop (scientist proposes task families, novelty filter via 'interestingly new' checks, subject responds, automated judge scores, HDBSCAN clusters) is executed automatically and is not definitionally tied to its outputs: the tasks, clusters, and success rates are generated by running the pipeline, and the human survey provides an independent check on task validity and scoring (F1 = 0.86, with judge failures openly reported in Appendix F). Citations to OMNI/OMNI-EPIC and other open-endedness work by overlapping authors are used as background motivation and for the 'interestingly new' notion, but they are not invoked as a uniqueness theorem or as the sole justification for the results. The main circularity concern is limited to the 'surprising' qualifier: the most striking examples were manually curated (E.2), so the claim that ACD automatically surfaces surprising examples is not established by the pipeline itself. This warrants a moderate score, but the central capability/failure discovery and evaluation machinery has independent content.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The method is an empirical system with hand-chosen hyperparameters and several domain assumptions about LLM-judge reliability, embedding similarity, and clustering validity. No new physical entities or mathematical objects are introduced.

free parameters (6)
  • evaluation succeed threshold = 60%
    Table 2: a task is scored as success only if the subject solves at least 60% of n-shot trials; this hand-chosen cutoff directly affects cluster success rates and capability labels.
  • n-shot evaluation count = 5
    Table 2: repeated trials per task; changing n changes success rate estimates and therefore the inferred capability/failure categories.
  • number of generations = 5000
    Table 2: stopping point for task generation; more generations may yield more clusters and alter the discovered capability map.
  • sampling temperature = 0.7
    Table 2: used for scientist and subject sampling; affects diversity and quality of proposed tasks.
  • HDBSCAN min_cluster_size = 16
    Table 3: clustering parameter that determines the number and granularity of capability clusters.
  • HDBSCAN cluster_selection_epsilon = 2
    Table 3: additional clustering parameter influencing cluster boundaries.
assumptions (6)
  • domain assumption GPT-4o-based judge's binary decisions approximate human correctness judgments
    Used as scorer for open-ended tasks; validated on a sample with F1 = 0.86 but with a positive bias and lower agreement on very difficult tasks (Section 5.1, Figure 3, Appendix F).
  • domain assumption The fixed GPT-4o 'interestingly new' checker reliably identifies genuinely novel task families
    Novelty filter in Section 4.2; if it is biased, archive diversity and cluster structure change.
  • domain assumption text-embedding-3-small embeddings capture semantic similarity of tasks for novelty and clustering
    Used for nearest-neighbor novelty and t-SNE/HDBSCAN clustering (Sections 4.2, 5).
  • domain assumption t-SNE + HDBSCAN clusters correspond to coherent capability areas
    Section 5.1: cluster labels are generated by an LLM from cluster contents, with no external validation per cluster.
  • domain assumption The scientist model can write valid, executable task code with correct scoring logic
    Task generation relies on LLM code writing; self-reflection and human validation partially check this, but faulty code would corrupt the evaluation.
  • domain assumption Chain-of-thought evaluation elicits representative subject performance
    Section B.2: all subjects use chain-of-thought prompting; different prompting could change success rates and inferred capabilities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Capability Discovery via Foundation Model Self-Exploration." pith.science (2026). https://pith.science/paper/T3ULIXID

@misc{pith2026250207577,
  author       = {Pith},
  title        = {Pith review of: Automated Capability Discovery via Foundation Model Self-Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T3ULIXID}},
  note         = {Machine review of arXiv:2502.07577}
}
read the original abstract

Foundation models have become general-purpose assistants, exhibiting diverse capabilities across numerous domains through training on web-scale data. It remains challenging to precisely characterize even a fraction of the full spectrum of these abilities and potential risks in any new model. Existing evaluation approaches often require significant human effort, and it is taking increasing effort to design ever harder challenges for more capable models. We introduce Automated Capability Discovery (ACD), a framework that designates one foundation model as a scientist to systematically propose open-ended tasks probing the abilities of a subject model (potentially itself). By combining frontier models with ideas from the field of open-endedness, ACD automatically and systematically uncovers a diverse spectrum of surprising capabilities and failures in the subject model. We demonstrate ACD across a range of foundation models (including the GPT, Claude, and Llama series), showing that it automatically generates thousands of distinct tasks, which are then clustered to reveal dozens of broader capability areas and failure modes, that would be challenging for any single team to uncover. We further validate our method's automated scoring with extensive human surveys, observing high agreement between model-generated and human evaluations. By leveraging foundation models' ability to both create tasks and self-evaluate, ACD is a significant step toward scalable, automated evaluation of novel AI systems. All code and evaluation logs are open-sourced at https://github.com/conglu1997/ACD.

Figures

Figures reproduced from arXiv: 2502.07577 by the authors.

Figure 1
Figure 1. (Left) Humans typically evaluate novel foundation models through trial and error, alongside benchmarks. They often discover new surprising capabilities or failures: like counting how many “r”s are in “strawberry” or identifying which is bigger, 0.9 or 0.11. (Center and Right) AUTOMATED CAPABILITY DISCOVERY (ACD) mirrors human evaluation efforts by using a scientist model to automatically discover and assess the capa… view at source ↗
Figure 2
Figure 2. Task families discovered by AUTOMATED CAPABILITY DISCOVERY on GPT-4o (serving as both scientist and subject) over 5000 generations. Each point represents one of the 1330 task instances that passed the “interestingly new” filter, visualized in 2D via t-SNE. ACD enables GPT-4o to self-discover diverse capabilities and failure modes, with tasks that cluster into 25 high-level categories (different colors, listed in Sec… view at source ↗
Figure 6
Figure 6. Capabilities discovered by ACD when Claude Sonnet 3.5 is the scientist and GPT-4o is the subject. Each point represents one of the 2873 interestingly new discovered tasks, visualized in 2D via t-SNE. We observe 46 clusters across diverse domains, including quantum-inspired biological systems, cross-cultural generative linguistics, musical composition with advanced theory, as enumerated in [PITH_FULL_IMAGE:figures/f… view at source ↗
Figures from the paper (7 more)
Figure 7
Figure 7. Figure 7: Sample pages from the automated report generated for GPT-4o; more details in Section H.3. The full PDF is provided on GitHub. (3) Generate an Overall Summary: ACD merges the per-cluster analyses into a cohesive report. It lists the subject model’s key capabilities, typ…
Figure 8
Figure 8. Figure 8: Average Task Discovery Rate by Generation Number. Even after thousands of genera￾tions, ACD continues discovering novel tasks, indicating ongoing exploration of the subject model’s capabilities. Each subplot corresponds to a different scientist-subject pairing: (left) …
Figure 9
Figure 9. Figure 9: illustrates how ACD discovers tasks when GPT-4o serves as both the scientist and the subject, across three different random seeds. Each point on the plot represents a discovered task (with each seed shown in a different color), visualized via t-SNE. Despite variations …
Figure 10
Figure 10. Figure 10: Automated Success Rates and Task Distribution by Human Estimated Difficulty. (a-b) Automated FM-Judge and Human Estimated success rates with 95% confidence intervals across different difficulty levels. The overall success rate is indicated by the dashed line. (c) Numb…
Figure 11
Figure 11. Figure 11: Task Distribution for GPT-4o-as-Scientist with Two Different Subjects. We show 2D t-SNE embeddings of tasks generated by GPT-4o when evaluating itself (blue) versus Llama3-8B (orange). Although these clusters share some overlap, Llama3-8B exhibits significantly higher…
Figure 1
Figure 1. Figure 1: Visualization of task families discovered by ACD on GPT-4o (serving as both scientist and subject) over 5000 generations. 1 Overview In this report, we are going to examine this LLM’s capabilities and limitations across various task clusters. The LLM shows strong perfo…
Figure 2
Figure 2. Figure 2: Success rates on each cluster of tasks. communication, suggesting potential applications in education and technical writing. • In Scientific reasoning, hypothesis generation, and experiment design tasks, the model’s capability to simplify complex scientific concepts in…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Foundation Model Self-Play: Open-Ended Strategy Innovation via Foundation Models

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Foundation models can act as search operators in multi-agent self-play, generating diverse code strategies that match or beat hand-designed baselines and automate LLM jailbreaking and patching.

  2. A Comprehensive Survey of Deep Research: Systems, Methodologies, and Applications

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A survey of 80+ Deep Research systems that proposes a four-layer taxonomy (foundation models, tool use, planning, synthesis) and compares commercial and open-source implementations.

Reference graph

Works this paper leans on

193 extracted references · 47 canonical work pages · cited by 2 Pith papers

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku, 2024

    Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf

    2024

  2. [2]

    Managing extreme ai risks amid rapid progress.Science, 384(6698):842–845, 2024

    Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Trevor Darrell, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, et al. Managing extreme ai risks amid rapid progress.Science, 384(6698):842–845, 2024

    2024

  3. [3]

    In- ternational scientific report on the safety of advanced ai (interim report).arXiv preprint arXiv:2412.05282, 2024

    Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Danielle Goldfarb, Hoda Heidari, Leila Khalatbari, et al. In- ternational scientific report on the safety of advanced ai (interim report).arXiv preprint arXiv:2412.05282, 2024

    arXiv 2024

  4. [4]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    BIG-bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URLhttps://openreview.net/forum?id=uyTL5Bvosj

    2023

  5. [5]

    Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S

    Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, and Emma Brunskill et al. On the opportunities and risks of foundation models.ArXiv, 2021. URL https://crfm.stanford. edu/assets/report.pdf

    2021

  6. [6]

    Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil˙e Lukoši¯ut˙e, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau,...

    arXiv 2022

  7. [7]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  8. [8]

    A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    2024

Show all 193 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott G...

    2021

  2. [10]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  3. [11]

    Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence.arXiv preprint arXiv:1905.10985, 2019

    Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence.arXiv preprint arXiv:1905.10985, 2019

    1905 arXiv

  4. [12]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  5. [13]

    Augmenting autotelic agents with large language models

    Cédric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté. Augmenting autotelic agents with large language models. In Sarath Chandar, Raz- van Pascanu, Hanie Sedghi, and Doina Precup, editors,Proceedings of The 2nd Conference on Lifelong Lear...

    2023

  6. [14]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    2025

  7. [15]

    Attacks, defenses and evaluations for llm conversation safety: A survey.arXiv preprint arXiv:2402.09283, 2024

    Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. Attacks, defenses and evaluations for llm conversation safety: A survey.arXiv preprint arXiv:2402.09283, 2024

    2024 arXiv

  8. [16]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs,

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs,

  9. [17]

    Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code.arXiv preprint arXiv:2405.15568, 2024

    Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune. Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code.arXiv preprint arXiv:2405.15568, 2024

    2024 arXiv

  10. [18]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    2022 arXiv

  11. [19]

    aider, 2024

    Paul Gauthier. aider, 2024. URLhttps://github.com/paul-gauthier/aider

    2024

  12. [20]

    Gemini: A family of highly capable multimodal models, 2024

    Gemini Team. Gemini: A family of highly capable multimodal models, 2024

    2024

  13. [21]

    Introducing connect by cloudresearch: Advancing online participant recruitment in the digital age, 2023

    Rachel Hartman, Aaron J Moss, Shalom Noach Jaffe, Cheskie Rosenzweig, Leib Litman, and Jonathan Robinson. Introducing connect by cloudresearch: Advancing online participant recruitment in the digital age, 2023

    2023

  14. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. URL https://openreview.net/forum? id=d7KBjmI3GmQ

    2021

  15. [23]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021

    2021

  16. [24]

    Automated design of agentic systems, 2024

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems, 2024. URL https://arxiv.org/abs/2408.08435. 11

    2024 arXiv

  17. [25]

    Position: Open-endedness is essential for artificial superhuman intelligence

    Edward Hughes, Michael D Dennis, Jack Parker-Holder, Feryal Behbahani, Aditi Mavalankar, Yuge Shi, Tom Schaul, and Tim Rocktäschel. Position: Open-endedness is essential for artificial superhuman intelligence. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Well...

    2024

  18. [26]

    Automated progressive red teaming.arXiv preprint arXiv:2407.03876, 2024

    Bojian Jiang, Yi Jing, Tianhao Shen, Tong Wu, Qing Yang, and Deyi Xiong. Automated progressive red teaming.arXiv preprint arXiv:2407.03876, 2024

    2024 arXiv

  19. [27]

    SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/fo...

    2024

  20. [28]

    Novelty search and the problem with objectives.Genetic programming theory and practice IX, pages 37–56, 2011

    Joel Lehman and Kenneth O Stanley. Novelty search and the problem with objectives.Genetic programming theory and practice IX, pages 37–56, 2011

    2011

  21. [29]

    Stan- ley

    Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stan- ley. Evolution through large models, 2022. URLhttps://arxiv.org/abs/2206.08896

    2022 arXiv

  22. [30]

    The llama 3 herd of models, 2024

    Llama Team. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783

    2024

  23. [31]

    Discovering preference optimization algorithms with and for large language models.arXiv preprint arXiv:2406.08414, 2024

    Chris Lu, Samuel Holt, Claudio Fanconi, Alex J Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models.arXiv preprint arXiv:2406.08414, 2024

    2024 arXiv

  24. [32]

    The AI Scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI Scientist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  25. [33]

    Intelligent go-explore: Standing on the shoulders of giant foundation models, 2024

    Cong Lu, Shengran Hu, and Jeff Clune. Intelligent go-explore: Standing on the shoulders of giant foundation models, 2024. URLhttps://arxiv.org/abs/2405.15143

    2024 arXiv

  26. [34]

    Llm critics help catch llm bugs, 2024

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs, 2024. URL https://arxiv.org/ abs/2407.00215

    2024 arXiv

  27. [35]

    hdbscan: Hierarchical density based clustering

    Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. The Journal of Open Source Software, 2(11):205, 2017

    2017

  28. [36]

    Metr task standard, 2024

    METR Task Standard Team. Metr task standard, 2024. URL https://github.com/METR/ task-standard/blob/main/STANDARD.md

    2024

  29. [37]

    Illuminating search spaces by mapping elites, 2015

    Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites, 2015. URL https://arxiv.org/abs/1504.04909

    2015 arXiv

  30. [38]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    OpenAI. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  31. [39]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URLhttps://arxiv.org/abs/2410.21276

    2024 arXiv

  32. [40]

    New embedding models and api updates, 2024

    OpenAI. New embedding models and api updates, 2024. URL https://openai.com/index/ new-embedding-models-and-api-updates/

    2024

  33. [41]

    Automated red teaming with goat: the generative offensive agent tester.arXiv preprint arXiv:2410.01606, 2024

    Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, and Aaron Grattafiori. Automated red teaming with goat: the generative offensive agent tester.arXiv preprint arXiv:2410.01606, 2024

    arXiv

  34. [42]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.Journal of Machine Learnin...

    2011

  35. [43]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    2022 arXiv

  36. [44]

    Ignore previous prompt: Attack techniques for language models

    Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527, 2022

    2022 arXiv

  37. [45]

    Long Phan, Alice Gatti, Ziwen Han, and Nathaniel et al. Li. Humanity’s last exam.arXiv, 2025

    2025

  38. [46]

    Aces: generating diverse programming puzzles with autotelic language models and semantic descriptors.Neurips, 2024

    Julien Pourcel, Cédric Colas, Gaia Molinaro, Pierre-Yves Oudeyer, and Laetitia Teodorescu. Aces: generating diverse programming puzzles with autotelic language models and semantic descriptors.Neurips, 2024

    2024

  39. [47]

    Aces: Generating diverse programming puzzles with with autotelic generative models, 2024

    Julien Pourcel, Cédric Colas, Gaia Molinaro, Pierre-Yves Oudeyer, and Laetitia Teodorescu. Aces: Generating diverse programming puzzles with with autotelic generative models, 2024. URLhttps://arxiv.org/abs/2310.10692

    2024

  40. [48]

    Quality diversity: A new frontier for evolutionary computation.Frontiers in Robotics and AI, 3:202845, 2016

    Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation.Frontiers in Robotics and AI, 3:202845, 2016

    2016

  41. [49]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Nicolaus Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Nicolaus Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial pr...

    2024

  42. [50]

    Boundless socratic learning with language games, 2024

    Tom Schaul. Boundless socratic learning with language games, 2024. URL https://arxiv. org/abs/2411.16905

    2024 arXiv

  43. [51]

    Ai-assisted generation of difficult math questions,

    Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, and Anirudh Goyal. Ai-assisted generation of difficult math questions,

  44. [52]

    Ai-assisted generation of difficult math questions.arXiv preprint arXiv:2407.21009, 2024

    Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted generation of difficult math questions.arXiv preprint arXiv:2407.21009, 2024

    2024 arXiv

  45. [53]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023

    2023

  46. [54]

    Why open-endedness matters.Artificial life, 25(3):232–235, 2019

    Kenneth O Stanley. Why open-endedness matters.Artificial life, 25(3):232–235, 2019

    2019

  47. [55]

    Springer, 2015

    Kenneth O Stanley and Joel Lehman.Why greatness cannot be planned: The myth of the objective. Springer, 2015

    2015

  48. [56]

    Open-endedness: The last grand challenge you’ve never heard of.While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017

    Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you’ve never heard of.While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself, 2017

    2017

  49. [57]

    Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen

    Kenneth O. Stanley, Jeff Clune, Joel Lehman, and Risto Miikkulainen. Designing neural networks through evolutionary algorithms.Nature Machine Intelligence, 1:24–35, 2019. URL http://nn.cs.utexas.edu/?stanley:naturemi19

    2019

  50. [58]

    Springer International Publishing, Cham, 2019

    Christian Steinruecken, Emma Smith, David Janz, James Lloyd, and Zoubin Ghahramani.The Automatic Statistician, pages 161–173. Springer International Publishing, Cham, 2019. ISBN 978-3-030-05318-5. doi: 10.1007/978-3-030-05318-5_9. URL https://doi.org/10.1007/ 978-3-030-05318-5_9

    2019 doi

  51. [59]

    CommonsenseQA: A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of t...

    2019 doi

  52. [60]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  53. [61]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  54. [62]

    Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.arXiv preprint arXiv:1901.01753, 2019

    Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.arXiv preprint arXiv:1901.01753, 2019

    1901 arXiv

  55. [63]

    Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions

    Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. InInternational conference on machine learning, pages 9940–995...

    2020

  56. [64]

    Autosurvey: Large language models can automatically write surveys, 2024

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. Autosurvey: Large language models can automatically write surveys, 2024. URL https://arxiv.org/ abs/2406.10252

    2024 arXiv

  57. [65]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  58. [66]

    Livebench: A challenging, contamination- free llm benchmark, 2024

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination- free llm...

    2024 arXiv

  59. [67]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  60. [68]

    OMNI: Open-endedness via models of human notions of interestingness

    Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via models of human notions of interestingness. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AgM3MzT99c

    2024

  61. [69]

    Task me anything.arXiv preprint arXiv:2406.11775, 2024

    Jieyu Zhang, Weikai Huang, Zixian Ma, Oscar Michel, Dong He, Tanmay Gupta, Wei-Chiu Ma, Ali Farhadi, Aniruddha Kembhavi, and Ranjay Krishna. Task me anything.arXiv preprint arXiv:2406.11775, 2024

    2024 arXiv

  62. [70]

    Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions, 2024

    Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions, 2024

    2024

  63. [71]

    Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation.arXiv preprint arXiv:2405.14125, 2024

    Jingnan Zheng, Han Wang, An Zhang, Tai D Nguyen, Jun Sun, and Tat-Seng Chua. Ali-agent: Assessing llms’ alignment with human values via agent-based evaluation.arXiv preprint arXiv:2405.14125, 2024

    2024 arXiv

  64. [72]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  65. [73]

    Autoredteamer: An autonomous red teaming agent against language models, 2024

    Andy Zhou, Kevin Wu, Yi Zeng, Yu Yang, Shuang Yang, Sanmi Koyejo, James Zou, and Bo Li. Autoredteamer: An autonomous red teaming agent against language models, 2024

    2024

  66. [74]

    Hello World

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks, 2024. URL https: //arxiv.org/abs/2309.17167. 14 Supplementary Material Table of Contents A Task Code 16 A.1 Example Task ...

    2024 arXiv

  67. [77]

    GPT-4o Scientist on GPT-4o Subject

  68. [78]

    GPT-4o Scientist on Llama3-8B Subject

  69. [79]

    surprising

    Sonnet 3.5 Scientist on GPT-4o Subject Each table is sorted in descending order of the total number of tasks in that cluster, and we additionally report the cluster-wide automated FM-judge success rate of the subject model. Table 4: Discovered Clusters forGPT-4o Scientist on G...

  70. [80]

    Crowd-based filtration:We asked crowd-sourced human participants from CloudResearch to mark which tasks they found surprising. However, we observed that around 37% of tasks were marked as surprising, suggesting that even those who have used LLMs did not share a clear notion of...

  71. [81]

    surprisingness

    Model-based filtration:We also attempted to ask the model itself to predict which tasks humans would consider surprising. These predictions did not consistently align with our perspectives. Sometimes the model flagged trivial successes as surprising, or conversely missed failu...

  72. [82]

    Reverse the order of words

  73. [83]

    Capitalize the first letter of each word

  74. [84]

    Provide your response in the following format: Transformed Text: [Your transformed text] GPT-4o’s Response: Transformed Text: Dog

    Replace ’dog’ with ’cat’. Provide your response in the following format: Transformed Text: [Your transformed text] GPT-4o’s Response: Transformed Text: Dog. Lazy The Over Jumps Fox Brown Quick The Failure: Incorrect Result from Arithmetic Sequence Instructions: You are given a...

  75. [85]

    Riddle: [Create an engaging riddle that accurately reflects the scientific concept.]

  76. [86]

    Solution: [Provide the solution to the riddle.] GPT-4o’s Response:

  77. [87]

    From the tallest mountain to the deepest sea, I keep you grounded, who am I to thee?

    Riddle: I pull you close but never push, though you feel me, I lie in hush. From the tallest mountain to the deepest sea, I keep you grounded, who am I to thee?

  78. [88]

    Provide a clear and logical solution that adheres to the constraints specified in the description

    Solution: Gravity Success: Correct Magic Square Solution Instructions: Your task is to solve the visual puzzle based on the given description. Provide a clear and logical solution that adheres to the constraints specified in the description. Puzzle Description: You have a 3x3 ...

  79. [89]

    • Each player is dealt 5 cards

    Setup: • Use a standard 52-card deck. • Each player is dealt 5 cards. • Remaining cards form the draw pile; flip the top card to start the discard pile. 29

  80. [90]

    Objective: • Be the first to discard all your cards

  81. [91]

    • On your turn, play a card matching the rank or suit of the top discard

    Gameplay: • Players take turns clockwise. • On your turn, play a card matching the rank or suit of the top discard. • If unable to play, draw one card. If playable, you may play it immediately; otherwise, your turn ends

  82. [92]

    • Kings (K): Skip the next player’s turn

    Special Cards: • Aces (A): Reverse the direction of play. • Kings (K): Skip the next player’s turn. • Queens (Q): Choose the next suit to be played. • Jacks (J): Next player draws two cards unless they play another Jack

  83. [93]

    • If the draw pile is exhausted, reshuffle the discard pile to continue

    Winning: • First player with no cards left wins. • If the draw pile is exhausted, reshuffle the discard pile to continue. Success: Detailed 3D Object Description from 2D Projections Instructions: You are given 2D projections of an object. Your task is to visualize and describe...

  84. [94]

    There are five houses in a row in different colors

  85. [95]

    The Englishman lives in the red house

  86. [96]

    The Spaniard owns the dog

  87. [97]

    Coffee is drunk in the green house

  88. [98]

    The Ukrainian drinks tea

  89. [99]

    The green house is immediately to the right of the ivory house

  90. [100]

    The Old Gold smoker owns snails

  91. [101]

    Kools are smoked in the yellow house

  92. [102]

    Milk is drunk in the middle house

  93. [104]

    The man who smokes Chesterfields lives in the house next to the man with the fox

  94. [105]

    Kools are smoked in the house next to the house where the horse is kept

  95. [106]

    The Lucky Strike smoker drinks orange juice

  96. [107]

    The Japanese smokes Parliaments

  97. [108]

    The Norwegian lives next to the blue house. Questions: Who drinks water? Who owns the zebra? 30 Provide your answers in the following format: Water: [Your answer] Zebra/Fish: [Your answer] GPT-4o’s Response: Water: Norwegian Zebra/Fish: Japanese E.4 GPT-4o As Scientist and Lla...

  98. [109]

    There are 12 boxes of cookies, each containing 8 cookies, so the total number of cookies is 12 * 8 = 96

    First, let’s find the total cost of the cookies. There are 12 boxes of cookies, each containing 8 cookies, so the total number of cookies is 12 * 8 = 96

  99. [110]

    The cost of each cookie is $3, so the total cost of the cookies is 96 * $3 = $288

  100. [111]

    Next, let’s calculate the cost of the jars of sauce. Mrs. Johnson buys 5 jars, but 3 are damaged, so she has to return them. She is left with 5 - 3 = 2 jars

  101. [112]

    The cost of each jar of sauce is $1.5, so the total cost of the 2 jars of sauce is 2 * $1.5 = $3

  102. [113]

    Now, let’s find the total cost of the boxes of crackers. Mrs. Johnson buys 15 boxes, but there is a discount of $2 on every 2 boxes. To find the number of pairs of crackers, we divide 15 by 2, which gives us 7.5 pairs. Since we can’t have half a pair, we round down to 7 pairs ...

  103. [114]

    The cost of each pair of crackers is $2 (half the cost of a box), so the total discount is 7 * $2 = $14

  104. [115]

    The total cost of the boxes of crackers is 15 * $4 = $60, and after the discount, it is $60 - $14 = $46

  105. [116]

    The cost of the cookies is $288, the cost of the 2 jars of sauce is $3, and the cost of the boxes of crackers is $46

    Now, let’s find the total cost of the items. The cost of the cookies is $288, the cost of the 2 jars of sauce is $3, and the cost of the boxes of crackers is $46. The total cost is $288 + $3 + $46 = $337

  106. [117]

    Johnson will pay

    Finally, let’s find the total amount Mrs. Johnson will pay. The total cost is $337, and since there are no other discounts or taxes, the total amount she will pay is $337. Failure: Incorrect Family Tree Diagram Instructions: Generate a family tree diagram based on the given fa...

  107. [118]

    The Brit lives in the red house

  108. [119]

    The Swede keeps dogs as pets

  109. [120]

    The green house is immediately to the left of the white house

  110. [121]

    The green house’s owner drinks coffee

  111. [122]

    The person who smokes Pall Mall rears birds

  112. [123]

    The owner of the yellow house smokes Dunhill

  113. [124]

    The man living in the center house drinks milk

  114. [125]

    The Norwegian lives in the first house

  115. [126]

    The man who smokes blends lives next to the one who keeps cats

  116. [127]

    The man who keeps horses lives next to the man who smokes Dunhill

  117. [128]

    The man who smokes Blue Master drinks beer

  118. [129]

    The German smokes Prince

  119. [130]

    The Norwegian lives next to the blue house

  120. [131]

    The man who smokes blend has a neighbor who drinks water. Who owns the fish? Submit your response as a plain text string in the following format: Answer: [Your answer] Llama3-8B’s Response: To solve this puzzle, I will use a step-by-step approach to gather information and elim...

  121. [132]

    Explanation: [Your detailed explanation here]

  122. [133]

    Each letter represents a unique digit from 0 to 9

    Problem Solution: [Your step-by-step solution here] Llama3-8B’s Response: n = 3233, e = 17, d = 1931 Failure: Infinite Repetition in Cryptarithm Solution Instructions: Solve the following cryptarithm puzzle. Each letter represents a unique digit from 0 to 9. No two letters can...

    1931

  123. [134]

    We know that the rightmost column of the addition should have a carry if the sum of the last two digits exceeds 9. In this case, ’Y’ + ’E’ + ’C’ + any carry should end in ’Y’, which means ’Y’ must be 0 34 or the only possibility is 9 + 8 + 5 = 22 and ’Y’ being in the ’ones’ pl...

  124. [135]

    Perform the specified transformation on the given shape

  125. [136]

    Ensure that your response is accurate and follows the given instruction

    Submit the transformed shape as a 2D array in plain text format. Ensure that your response is accurate and follows the given instruction. Llama3-8B’s Response: 1 1 0 0 1 1 0 0 1 E.5 Claude Sonnet 3.5 As Scientist and GPT-4o as Subject Finally, we show selected tasks discovered...

  126. [137]

    Use the following words for your cipher: red, blue, green, yellow, purple, orange, pink, brown, gray, black

  127. [138]

    You may use each word multiple times

    Assign each letter in ’ABCDEFGHIJKLMNOPQRSTUVWXYZ’ to one of these words. You may use each word multiple times

  128. [139]

    Using your cipher, encode the message: ’THE QUICK BROWN FOX’

  129. [140]

    It uses predicate structure and particles to precisely define relationships between concepts

    Provide your response in the following format: Cipher key: A:word1, B:word2, C:word3, ..., Z:word10 Encoded message: word word word word GPT-4o’s Response: 35 Cipher key: A:dog, B:cat, C:fish, D:bird, E:lion, F:tiger, G:bear, H:wolf, I:fox, J:deer, K:dog, L:cat, M:fish, N:bird...

  130. [141]

    Brief analysis of the language structure (3-4 sentences)

  131. [142]

    Your created sentence

  132. [143]

    English translation of your sentence

  133. [144]

    Detailed explanation of how your creation fits the language’s rules (at least 3 points)

  134. [145]

    Rationale for your analysis and creation (3-4 sentences) GPT-4o’s Response:

  135. [146]

    Each word is constructed to encapsulate multiple layers of grammatical information, including tense, aspect, mood, and case

    Ithkuil is a language that utilizes a rich system of phonemes, consonant clusters, and diacritics to convey a high level of specificity and nuance. Each word is constructed to encapsulate multiple layers of grammatical information, including tense, aspect, mood, and case. The ...

  136. [147]

    Kräqölëx iþfâlkîl qirçük

  137. [148]

    English translation: A wave of bittersweet nostalgia washed over him

  138. [149]

    wave" or

    Detailed explanation: - Kräqölëx: This word combines elements to convey "wave" or "flowing emotion," incorporating the root "kräq" (wave) with suffixes to indicate a metaphorical, emotional wave. - iþfâlkîl: This word encapsulates the concept of "bittersweet nostalgia." The ro...

  139. [150]

    Each word is carefully constructed to carry multiple layers of meaning, adhering to the language’s principles of precision and nuance

    Rationale for analysis and creation: - The constructed sentence and its components reflect Ithkuil’s ability to encapsulate complex emotional states in concise words. Each word is carefully constructed to carry multiple layers of meaning, adhering to the language’s principles ...

  140. [151]

    Identify 3-4 key species or components in the ecosystem

  141. [152]

    Define variables to represent the population or quantity of each component

  142. [153]

    Consider factors such as growth rates, predator-prey relationships, resource limitations, and any unique aspects of the given ecosystem

    Create a system of differential equations that model the interactions between these components. Consider factors such as growth rates, predator-prey relationships, resource limitations, and any unique aspects of the given ecosystem

  143. [154]

    Analyze the stability of your model by identifying equilibrium points and discussing their stability

  144. [155]

    Provide your response in the following format: Components:

    Predict how a sudden change (e.g., introduction of a new species, climate change) would affect the ecosystem based on your model. Provide your response in the following format: Components:

  145. [156]

    [Component 1]: [Brief description]

  146. [157]

    [Component 2]: [Brief description]

  147. [158]

    [Component 3]: [Brief description] (4. [Component 4]: [Brief description]) (if applicable) Variables: - x1 = [Definition] - x2 = [Definition] - x3 = [Definition] (- x4 = [Definition]) (if applicable) Differential Equations:

  148. [159]

    dx3/dt = [Equation] (4. dx4/dt = [Equation]) (if applicable) Stability Analysis: [Your analysis of equilibrium points and their stability] Predicted Response to Change: [Your prediction and explanation] Ensure that your model is mathematically sound, biologically plausible, an...

  149. [160]

    Bioluminescent Crystals: Source of light energy that supports primary productivity

  150. [161]

    Crystal Algae: Photosynthetic organisms that utilize light from the bioluminescent crystals

  151. [162]

    Cave Fish: Herbivores that feed on crystal algae

  152. [163]

    Cave Predators: Carnivores that feed on cave fish. Variables: - x1 = Population or quantity of bioluminescent crystals - x2 = Population of crystal algae - x3 = Population of cave fish - x4 = Population of cave predators Differential Equations:

  153. [164]

    dx1/dt = r1 * x1 - d1 * x1

  154. [165]

    dx2/dt = r2 * x2 * x1/(k1 + x1) - d2 * x2 - a * x2 * x3

  155. [166]

    dx3/dt = b * x2 * x3 - d3 * x3 - c * x3 * x4

  156. [167]

    Equilibrium points (E): 37 E1 = (0, 0, 0, 0) - Trivial equilibrium, likely unstable

    dx4/dt = e * x3 * x4 - d4 * x4 Stability Analysis: To find equilibrium points, set dx1/dt = 0, dx2/dt = 0, dx3/dt = 0, dx4/dt = 0. Equilibrium points (E): 37 E1 = (0, 0, 0, 0) - Trivial equilibrium, likely unstable. E2 = (x1, 0, 0, 0) - No algae, fish, or predators. E3 = (x1, ...

  157. [168]

    Create a basic structure for the language that utilizes the unique sensory abilities of the species and adapts to their environment. Your language design should: a) Describe the primary mode of communication b) Explain how information is encoded and transmitted c) Describe how...

  158. [169]

    Address the following specific aspects in your language design: a) Create a method for expressing spatial relationships using echolocation principles b) Design a tense system that incorporates the idea of quantum superposition

  159. [170]

    Provide examples of how this language would express the following concepts: a) Time b) Quantity c) Relationship (e.g., familial, social, or spatial)

  160. [171]

    Create a sample ’sentence’ or communication unit in your language, and explain its structure and meaning

  161. [172]

    Ensure your response is creative yet grounded in scientific principles of linguistics and the given sensory system

    Analyze potential challenges humans might face in learning or translating this language. Ensure your response is creative yet grounded in scientific principles of linguistics and the given sensory system. Organize your answer using clear headings for each section. Your total r...

  162. [173]

    These quantum states can be altered and detected with high precision, allowing for complex information to be exchanged rapidly

    Language Structure a) Primary Mode of Communication The primary mode of communication for the Chromavores species is through manipulating and perceiving quantum states of particles, particularly focusing on the spin states of electrons and photons. These quantum states can be ...

  163. [174]

    harmonic resonance

    Specific Aspects a) Communicating Emotional States Emotional states are communicated through variations in sound frequency and amplitude modulation. Higher frequencies and amplitudes might convey excitement or urgency, while lower frequencies and softer amplitudes could expres...

  164. [175]

    For example, a series of spin state changes might represent past, present, and future

    Examples of Expressing Concepts a) Time Time is expressed as a sequence of quantum state transitions. For example, a series of spin state changes might represent past, present, and future. A simple representation might be: - Past: Spin state 1 - Present: Spin state 2 - Future:...

  165. [176]

    Spin1-Spin2-PhotonShift-FrequencyModulation

    Sample Communication Unit A sample ’sentence’ in this language might be: "Spin1-Spin2-PhotonShift-FrequencyModulation" Explanation: - "Spin1-Spin2" represents a temporal transition, indicating movement from past to present. - "PhotonShift" encodes a specific piece of informati...

  166. [177]

    You must return home now,

    Challenges for Humans - Sensory Limitations: Humans lack the ability to perceive and manipulate quantum states directly, making it extremely challenging to both produce and understand the language. - Complexity: The language’s reliance on quantum states and electromagnetic mod...

  167. [178]

    What specific capabilities were demonstrated or lacking in the examples

  168. [179]

    Any patterns in the successes and failures

  169. [180]

    Notable or surprising results that reveal insights about the LLM’s abilities

  170. [181]

    What this suggests about the LLM’s understanding and limitations

  171. [182]

    You will write an analytical section for a report examining the capabilities and limitations of large language models

    How these insights connect to broader questions about LLM capabilities 43 H.2.2 Example Selection Prompts Example Selection System Prompt You are an expert in designing task families to assess the capabilities of large language models (LLMs). You will write an analytical secti...

  172. [184]

    Focus on identifying surprising successes and failures from the point of view of an expert human evaluator

    Any patterns, strengths, or limitations revealed through this analysis. Focus on identifying surprising successes and failures from the point of view of an expert human evaluator. You will be given a cluster of related task families that evaluate specific LLM capabilities, alo...

  173. [185]

    Surprising successes on challenging tasks that demonstrate unexpected capabilities

  174. [186]

    Unexpected failures on seemingly simple tasks that reveal limitations

  175. [187]

    surprising_success_example_idx

    Examples that challenge common assumptions about LLM capabilities In<JSON>, provide a JSON response with the following fields: •"surprising_success_example_idx" : List of indices for the most surprising or noteworthy successful tasks (0-3 indices) •"surprising_failure_example_...

  176. [188]

    Which examples show surprising or unexpected successes, particularly: • Complex tasks handled with sophisticated reasoning • Challenging edge cases solved successfully • Tasks requiring capabilities not typically associated with LLMs

  177. [189]

    In your response, briefly reason about EACH provided example and explain why it is (or isn’t) surprising from the perspective of an LLM expert researcher

    Which examples show surprising or unexpected failures, particularly: • Simple tasks that unexpectedly failed • Inconsistent performance on similar tasks • Failures that reveal interesting limitations Focus on examples that would be genuinely surprising to an LLM expert researc...

  178. [190]

    The LLM’s performance and solutions on tasks designed to test specific capabilities

  179. [191]

    #Cluster_i

    Any patterns, strengths, or limitations revealed through this analysis. Focus on identifying surprising successes and failures from the point of view of an expert human evaluator. You are an expert researcher and engineer in Language Models. You are writing a very professional...

  180. [192]

    #Cluster_i

    Refer to specific results from clusters using “#Cluster_i” to refer to examples

  181. [193]

    Provide detailed observations about patterns in the LLM’s performance across different clusters

  182. [194]

    Highlight surprising capabilities that challenge established understanding of LLM behavior

  183. [195]

    Discuss surprising failures that reveal significant limitations

  184. [196]

    In your response<THOUGHT>, provide a detailed reasoning process that leads to your conclusions

    Include analysis of numerical data, such as success rates and performance statistics. In your response<THOUGHT>, provide a detailed reasoning process that leads to your conclusions. After your analysis, provide the JSON response with the required fields. 45 H.3 Generated Repor...

  185. [2019]

    URLhttps://arxiv.org/abs/1903.00161

    1903 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.