Artificial Scientific Discovery

Antonio Norelli

REVIEW 2 major objections 6 minor 30 references

Artificial Scientific Discovery

T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An artificial scientist must learn to interpret symbols, not inherit an interpreter.

desk verdict A well-written thesis compiling four solid papers whose central 'artificial scientist' claim outruns the evidence, mainly because the flagship Odeen result assumes a favorable, pre-given data set. read the letter →

arxiv 2411.11672 v2 pith:ENKL4QJ6 submitted 2024-11-18 cs.AI cs.LG

Antonio Norelli This is my paper · ORCID

classification cs.AIcs.LG

keywords artificialscientistexplanatorylearningsymbolinterpretationCriticalRationalistNetworksOdeenbenchmarkmultimodalalignmentzero-shotclassificationlargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Across three connected studies, the thesis tries to establish what a machine needs in order to make original, communicable scientific discoveries. Its central claim is that the missing piece is autonomous symbol interpretation: the machine must learn the mapping from explanation strings to observations from paired examples, rather than being given a rigid interpreter for its explanation language. The empirical anchor is the Odeen benchmark, a small universe of 117,649 structures and roughly 24,794 grammar-generated rules, where Critical Rationalist Networks discover the correct explanation for 77.7% of unseen phenomena using only 32 labelled observations per phenomenon, against 22.5% for the best empiricist baseline. The thesis goes on to argue that the same interpretation problem appears in multimodal models and in large language models, whose chance-level performance on the Symbol Interpretation Task marks the largest human-machine gap in a large multi-model benchmark study.

What carries the argument

The load-bearing mechanism is the learned interpreter, a transformer encoder that maps a rule string and a structure to a label, trained only on paired explanation-observation data. Around it, a Critical Rationalist Network adds a stochastic Conjecture Generator that proposes candidate explanations from the 32 observations of a new phenomenon; at test time the network generates t conjectures, scores each with the interpreter on the labelled observations, and tags unseen structures with the best-scoring explanation. The second machinery, used for multimodal interpretation, is the relative-representation 'as if' step: represent an image by its similarities to a set of image-text pairs and then treat that vector as if it were the representation of the image's ideal caption, which aligns frozen unimodal encoders without training. The Odeen environment supplies the testbed: six-slot structures over a small alphabet, a grammar of roughly 24,794 rules, and a semantics encoded by a hardcoded interpreter that the learner is never allowed to use.

What would settle it

Run the Odeen scientist problem twice on the same test phenomena, once with the standard representative 32-observation sets and once with sets deliberately chosen so that each D0 is ambiguous between the target rule and at least one other rule in the grammar; if Nearest Rule Score stays near 0.777 in the ambiguous condition, the representativity premise is not load-bearing, and if it collapses toward the empiricist baseline, the discovery claim depends on the favourable starting point.

Watch

Extended reading notes

Core claim

The central claim is that an artificial scientist is possible only when the machine can acquire an interpreter for the language of explanations from data. In the Explanatory Learning formulation, explanations are free strings with no grammar or interpreter supplied, and the learner must build the map from symbols to phenomena out of explanation-observation pairs. The thesis reports that on Odeen, Critical Rationalist Networks—a Conjecture Generator paired with a learned Interpreter—achieve a Nearest Rule Score of 0.777 on 1,132 unseen phenomena with 32 observations each, whereas the strongest end-to-end empiricist model reaches only 0.225 and rarely produces a usable explanation (R-Acc 0.035). It further reports that a learned interpreter can beat the hardcoded ground-truth interpreter in a CRN, because it tolerates ambiguous or ungrammatical conjectures, and that large language models scored no better than random chance on the companion Symbol Interpretation Task.

Load-bearing premise

The load-bearing premise is that the learner is handed 32 labelled observations of the new phenomenon that are guaranteed to distinguish it from every other explainable phenomenon, rather than having to find or design those observations itself.

Editorial extensions

If this is right

Separating conjecture generation from conjecture testing by a learned interpreter gives a concrete architecture for few-shot scientific explanation, with time-varying computation controlled by the number of generated conjectures.
A learned interpreter can outperform the ground-truth one because it assigns meaning to malformed hypotheses; discovery is therefore not tied to syntactic correctness of the explanation language.
Multimodal understanding comparable to contrastively trained models can be assembled from frozen unimodal encoders plus a small set of coupled pairs, making interpretation editable and transparent—adding or removing knowledge is just adding or deleting stored embeddings.
Because large language models fail the Symbol Interpretation Task at random chance while humans solve it, scaling current architectures is not by itself a path to the symbol malleability an artificial scientist needs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The thesis's 77.7% discovery rate assumes the 32 initial observations are representative, i.e., they distinguish the target phenomenon from every other language-explainable phenomenon; the thesis itself flags that it does not model how the scientist actively acquires those observations, so transfer to uncurated settings is the main open risk.
A direct extension would turn the scientist problem into an active-learning loop: let a CRN choose which structures to observe and test whether the discovery rate rises or the required number of observations falls; this would also operationalize curiosity, which the conclusion names as missing from current models.
ASIF's success suggests that much of 'interpretation' in multimodal models is stored in the training pairs rather than in the weights, predicting that ASIF performance tracks the semantic distance between coupled dataset and downstream task—a measurable quantity.
The Symbol Interpretation Task offers a cheap falsifier for claims about LLM reasoning: rerun it on newer models; the thesis's position implies they should still perform at chance unless they incorporate an explicit multi-step uncertainty or symbol-redefinition mechanism.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: this is a PhD thesis that assembles four already-published papers (Olivaw, Explanatory Learning, ASIF, and the Big-Bench SIT task) into a narrative about building an artificial scientist. The individual chapters are credible and mostly well-executed; the overarching claim that autonomous symbol interpretation is the missing capability is programmatic and overreaches the evidence presented.

What is new here is the framing, not the results. Relative to the cited papers, the thesis adds an integrative story: from Olivaw's silent competence, to the EL formalization, to ASIF's transparent interpreter, to the SIT result that LLMs fail to interpret novel symbols. The EL framework and the Odeen benchmark are genuinely useful formalizations, and the CRN-versus-empiricist comparison (77.7% vs 22.5% NRS) is a striking demonstration that a rationalist approach can beat end-to-end learning when the language is learnable. ASIF is a strong cheap baseline, and SIT is a real contribution to Big-Bench. Credit is also due for the unusually clear writing and for the explicit acknowledgment of limitations.

The soft spots are real but not fatal. The most important is flagged by the author himself in Sec 3.7: the "scientist problem" assumes a representative D0 of 32 observations per phenomenon, with a guarantee that it disambiguates the target rule. The 77.7% discovery rate is measured under this deliberately favorable sampling (Appendix A.1 confirms the first 10 structures are chosen as near-miss pairs). The thesis does not model how the scientist actively obtains D0, which is the essence of experiment design in real science. So the central "artificial scientist" conclusion rests on a narrower demonstration than the rhetoric suggests. Second, Table 3 has no error bars or repeated seeds; this is addressable, but a reviewer would want them. Third, ASIF's hyperparameters were tuned on ImageNet validation; this is disclosed and the appendix shows mild sensitivity, but it slightly undercuts the "no training" framing.

Who this is for: readers interested in explainable AI, rationalist versus empiricist learning, or benchmarks for symbol grounding. It deserves a serious referee—probably as a monograph or a position piece—but the author should be pushed to either temper the artificial-scientist claim or add experiments with active data acquisition. I would not cite the thesis itself; I would cite the underlying papers.

Referee Report

2 major / 6 minor

Summary. This PhD thesis argues that realizing an artificial scientist requires a machine to learn its own interpreter—the mapping between explanatory symbols and observations—rather than relying on a fixed, human-provided interpreter. The argument is developed across four empirical chapters: (1) Olivaw, an AlphaGo-Zero-like Othello agent that reaches world-class play with minimal search but cannot communicate its knowledge; (2) the Explanatory Learning (EL) framework, the Odeen benchmark, and Critical Rationalist Networks (CRNs), which pair a conjecture generator with a learned interpreter and report 77.7% Nearest Rule Score on 1,132 unseen phenomena from 32 observations each, versus 22.5% for the best empiricist baseline (Table 3); (3) ASIF, a training-free procedure that couples frozen unimodal encoders with roughly 1.6M image-text pairs, reaching CLIP-competitive zero-shot accuracy at a fraction of the multimodal data budget; and (4) the Big-Bench Symbol Interpretation Task, on which LLMs score at chance while the best human raters reach perfection. The thesis concludes that autonomous symbol interpretation is the keystone open problem for machine-assisted scientific discovery.

Significance. If the results hold, the manuscript makes a substantive contribution: it articulates Explanatory Learning as a distinct formalization (explanation-observation pairs with no given interpreter), releases a parametric benchmark (Odeen) with a controlled compositional split (rules containing 'exactly two' are withheld from training and tested), and shows on that benchmark that a two-stage rationalist architecture beats end-to-end empiricist baselines of similar size. The finding that a learned interpreter can match or beat the ground-truth hardcoded interpreter on Odeen (Table 4) is genuine, non-obvious, and directly supports the thesis's central intuition. ASIF provides a simple, reproducible, transparent baseline for multimodal alignment with a disclosed data budget, and the SIT results are a clean, falsifiable negative result for LLM symbol interpretation. The manuscript ships unusually complete supporting material: training hyperparameters for Olivaw, cost analyses for CRNs (Tables 9-10), backbone specifications for ASIF, and released code for the main claims.

major comments (2)

[Sec. 3.2.1, Sec. 3.7, Appendix A.1] The Odeen scientist problem is defined by two guarantees (Sec. 3.2.1): P0 is explainable in L, and D0 is representative for P0, i.e., every other explainable phenomenon differs from P0 on at least one item of D0. This representativity guarantee is load-bearing: the CRN's test-time selection (Sec. 3.4.1) picks the conjecture with the highest hit rate on D0, and that procedure is well-posed precisely because D0 pins down P0 uniquely among the roughly 24k rules. The headline result of Table 3 (CRN NRS 0.777 vs. Emp-C 0.225) is therefore measured under a favorable data-generation protocol: Appendix A.1 states that the first 10 structures of each D0 are chosen by searching for near-miss pairs (similar structures, different labels) and the remaining 22 are selected to ensure the lack of ambiguity on the board. No sensitivity analysis is reported: with a uniformly random D0 of size 32, the representativity guarantee will generally fail (many rules agree with any 32 observations), and the measured discovery rate should be expected to drop, possibly sharply. Because the thesis's central claim is inferred from the magnitude of the CRN advantage, this point is load-bearing, and the thesis itself concedes in Sec. 3.7 that acquisition of D0 is not modeled. I ask the authors to either (a) add experiments that degrade the D0 guarantee (uniformly random D0, smaller k, noisy labels) and report CRN-vs-empiricist performance under those conditions, or (b) visibly narrow the claims in the Abstract and Introduction to the conditional setting in which representative observations are supplied. This is a scope/external-validity concern, not an internal inconsistency; the EL formalism itself is coherent.
[Abstract, Sec. 1.1, Sec. 6.1] The keystone claim is stated unconditionally: 'a true artificial scientist can only emerge when a machine can autonomously interpret symbols' (Sec. 1.1), and the Abstract presents the Odeen success as establishing it. The evidence actually supports a conditional statement: within the EL framework on the Odeen benchmark, a learned interpreter matches or slightly outperforms the ground-truth hardcoded interpreter (Table 4, 0.813 vs. 0.801 at n = 1438 rules), and CRNs beat the empiricist baselines (Table 3) — with the caveat that the learned-versus-hardcoded advantage reverses at n = 500 rules (Table 4: 0.354 vs. 0.377 at 10K structures), so the benefit is regime-dependent. No experiment in the thesis connects the Odeen results to the ASIF or SIT results in a way that tests whether interpreter learning is the limiting factor in those settings; the inference from three independent toy-domain results to 'the central open problem of machine-assisted scientific discovery' is an inductive leap. I recommend reformulating the keystone claim to be conditional on the EL setting and on the representative-D0 assumption raised in the previous comment, or extending the evidence to at least one environment with substantially different structure so that the universality claim is tested rather than asserted.

minor comments (6)

[Abstract and Sec. 5.2] The Abstract says SIT is 'fully solved by humans,' but Table 6 reports an average human score of 36.8% (chance 20%) with only the best raters reaching 100%; suggest 'solved by the best human raters, with average human performance well above chance.'
[Sec. 5.2 / Table 6] The claim that SIT shows 'the largest performance gap between human participants and LLMs' should specify that this holds for gaps in favor of humans, since several Big-Bench Lite tasks in the same table show larger absolute gaps in favor of models (e.g., auto_debugging: PaLM 540B 38.2 vs. human 13.7).
[Table 3 caption] Typo: 'hradcoded' should read 'hardcoded'; the same error appears in the prose accompanying the table.
[Sec. 4.4.2 / Table 5] The disclosure that k = 800 and p = 8 were tuned on a subset of the ImageNet validation set appears only in a table footnote; it deserves a sentence in the main text, with a pointer to the robustness analysis in Appendix B.3.
[Sec. 3.5 / Table 3] The Odeen results are reported without variance estimates; since the training sampler and the conjecture generator are stochastic, please report means and standard deviations over at least a few training seeds for the central Table 3/Table 8 comparisons.
[Throughout] Several spacing and typographical artifacts should be cleaned up, e.g., 'O livaw', 'T ransformer', 'VITb16', 'mutimodal' (Sec. 4.5.4), and 'sensibility' in the Appendix B.3 heading, which should be 'sensitivity.'

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the Odeen discovery results are empirical, and the representative-D0 guarantee is an acknowledged scope limitation rather than an input that forces the 77.7% outcome.

full rationale

The thesis is a narrative compilation of the author's prior publications, so self-citation is pervasive; however, none of the load-bearing quantitative claims reduces to its own inputs by construction. The Odeen 'scientist problem' (Sec. 3.2.1) assumes a representative set D0, and the paper explicitly acknowledges in Sec. 3.7 that it 'assumes a representative set of observations D0 to be given' and leaves the active discovery of D0 to future work. This is a scope limitation, not circularity: the guarantee makes the benchmark well-posed, but it does not force the CRN result, because the empiricist baselines with the same D0 reach only 22.5% NRS (Table 3). The CRN must still learn the interpreter from explanation-observation pairs and generate and verify conjectures; nothing in the definitions makes the interpreter I or the conjecture generator CG output the correct e0 automatically. The 'learned interpreter beats hardcoded interpreter' finding (Table 4) is likewise empirical rather than entailed by the EL formalism. The ASIF chapter builds on relative representations from prior work by the same group, but it explicitly relates its construction to kernel methods and k-NN (Secs. 4.2.3 and 4.2.6) and evaluates against external CLIP/LiT baselines on standard datasets. The SIT results come from the collaborative Big-Bench benchmark with human raters. No equation is simultaneously the input and the output of a claimed prediction, no fitted parameter is renamed as a discovery, and no uniqueness theorem from the authors' own prior work is invoked to exclude alternatives. The representativity assumption and the in-house Odeen benchmark lower external validity but do not constitute circular reasoning.

Assumptions & free parameters 4 free parameters · 4 assumptions · 4 invented entities

The thesis introduces a nested stack of self-created frameworks and benchmarks. The main empirical demonstrations of the EL framework happen on Odeen, an environment designed by the authors, while ASIF is validated on external standard datasets. Free parameters are modest and disclosed, but the Odeen design choices and the representativity assumption are load-bearing for the central claims.

free parameters (4)

Odeen observation budget k = 32
Number of labelled structures provided per new phenomenon in the scientist problem; chosen by hand, and the reported CRN discovery rates depend on it.
ASIF sparsity k = 800
Number of top similarity dimensions kept in relative representations; tuned on the ImageNet validation set (Table 5, footnote).
ASIF exponent p = 8
Nonlinearity applied to retained similarities; tuned jointly with k on the ImageNet validation set.
Odeen grammar and universe size = 6 positions, 7 piece types, 24,794 rules
Environment design choices that define the difficulty of the EL benchmark; these choices shape all CRN versus empiricist comparisons.

assumptions (4)

domain assumption EL assumes a representative set of observations D0 is given for each new phenomenon
Stated in Sec 3.7 as a limitation: the framework does not model active data acquisition. The representativity definition in Sec 3.2.1 guarantees D0 distinguishes P0 from other explainable phenomena, which is a strong premise.
domain assumption Odeen rules are generated by a fixed context-free grammar and interpreted by a hardcoded regular-expression interpreter
The benchmark is a closed formal world; the 'unknown language' is in fact a known formal language to the experimenters, which limits transfer of EL results to real scientific settings.
domain assumption ASIF's key premise that captions of similar images are themselves similar
The entire cross-modal alignment rests on this empirical regularity (Sec 4.1, Fig 18); it is not proven and may fail for niche or out-of-distribution domains.
domain assumption There is no science beyond human language
Prologue and Chapter 1; this philosophical stance defines the thesis's goal of explainability to humans and is not empirically established.

invented entities (4)

Explanatory Learning (EL) problem
purpose: Formalizes the task of learning an interpreter for explanations from observation-explanation pairs; presented as the keystone of the thesis.
Introduced by the authors; being a problem formulation, it has no external falsifiable handle beyond the Odeen benchmark they also created.
Odeen benchmark
purpose: A miniature scientific-discovery environment with a formal language and hidden rules.
Constructed by the authors; results on it are internal to the framework and have no independent external validation.
Critical Rationalist Networks (CRNs) independent evidence
purpose: A two-network architecture (conjecture generator plus learned interpreter) for solving EL problems.
The method is specified in enough detail to be reimplemented, and the authors provide implementation details; however, no independent replication is cited within the thesis.
ASIF procedure independent evidence
purpose: Training-free construction of a multimodal common space from unimodal encoders and coupled data.
The thesis notes subsequent use of ASIF by other teams in audio (Wang et al. 2023) and provides code, giving an external foothold.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Artificial Scientific Discovery." pith.science (2026). https://pith.science/paper/ENKL4QJ6

@misc{pith2026241111672,
  author       = {Pith},
  title        = {Pith review of: Artificial Scientific Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENKL4QJ6}},
  note         = {Machine review of arXiv:2411.11672}
}

read the original abstract

Rooted in the explosion of deep learning over the past decade, this thesis spans from AlphaGo to ChatGPT to empirically examine the fundamental concepts needed to realize the vision of an artificial scientist: a machine with the capacity to autonomously generate original research and contribute to the expansion of human knowledge. The investigation begins with Olivaw, an AlphaGo Zero-like agent that discovers Othello knowledge from scratch but is unable to communicate it. This realization leads to the development of the Explanatory Learning (EL) framework, a formalization of the problem faced by a scientist when trying to explain a new phenomenon to their peers. The effective EL prescriptions allow us to crack Zendo, a popular board game simulating the scientific endeavor. This success comes with a fundamental insight: an artificial scientist must develop its own interpretation of the language used to explain its findings, and not rely on a rigid existing interpreter. Questioning the very process of learning an interpreter, we turn our attention to the inner functioning of modern multimodal models. This culminates in a simple idea to build CLIP-like models where interpretation and perception are explicitly disentangled: a cost-effective approach that couples two unimodal models using little multimodal data and no further training. Finally, we discuss what ChatGPT and its siblings are still missing to become artificial scientists, and introduce the Big-Bench Symbol Interpretation Task, a benchmark about interpreting Zendo-like explanations that sees LLMs going no further than random chance while being instead fully solved by humans.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 20 canonical work pages

[2]

Bootstrapping Parallel Anchors for Relative Representations

Citeseer. Cannistraci, Irene, Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà ( 2023). “Bootstrapping Parallel Anchors for Relative Representations.” In: Tiny paper at ICLR

work page 2023
[5]

a satellite image of [CLASS NAME]

achieved a zero-shot classification score of 29.4%. While significantly better than random chance, this modest performance is not surpris- B.2 additional details on the eurosat experiment . 89 Figure 28: ASIF performance does not saturate earlier with smaller encoders. Classification accuracy keeps growing without satu- rating but is lower for smaller mod...

work page 2021
[6]

Beyond the Imitation Game: Quantify- ing and extrapolating the capabilities of language models

5.2.1 Big-Bench Big Bench serves as a comprehensive benchmark to evaluate and un- derstand the capabilities and limitations of contemporary language models (Srivastava et al. 2023). Developed collaboratively by 450 au- thors from 132 institutions, including myself, the benchmark consists of 204 diverse tasks covering areas such as linguistics, math, commo...

work page 1972
[8]

DeepCoder: Learning to write programs

url: https://www.anthropic.com/index/introducing-claude. Balog, M, AL Gaunt, M Brockschmidt, S Nowozin, and D Tarlow (2017). “DeepCoder: Learning to write programs.” In: 5th Interna- tional Conference on Learning Representations, ICLR 2017-Conference Track Proceedings. Basu, Samyadeep, Phil Pope, and Soheil Feizi (2021). “Influence Func- tions in Deep Lea...

2017
[9]

Relative rep- resentations enable zero-shot latent space communication

McGraw-hill New York. isbn: 0070428077. Mitchell, Tom M ( 1980). The need for biases in learning generalizations . Department of Computer Science, Laboratory for Computer Sci- ence Research. Moschella, Luca, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà ( 2022). “Relative rep- resentations enable zero-shot late...

work page 1980
[13]

Performance analysis of google colaboratory as a tool for accelerating deep learning ap- plications

Carneiro, Tiago, Raul Victor Medeiros Da Nóbrega, Thiago Nepomu- ceno, Gui-Bin Bian, Victor Hugo C De Albuquerque, Pedro Pe- drosa Reboucas Filho, and X X ( 2018). “Performance analysis of google colaboratory as a tool for accelerating deep learning ap- plications.” In: IEEE Access 6, pp. 61677–61685. bibliography 95 Caron, Mathilde, Hugo Touvron, Ishan M...

arXiv 2018
[14]

Dreamcoder: Growing gener- alizable, interpretable knowledge with wake-sleep bayesian pro- gram learning

Eco, Umberto (2000). Kant and the platypus: Essays on language and cog- nition. HMH. Ellis, Kevin, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum ( 2020). “Dreamcoder: Growing gener- alizable, interpretable knowledge with wake-sleep bayesian pro- gram learning.” In: arXi...

arXiv 2000
[17]

Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

MIT Technology Review. url: https://www.technologyreview.com/ 2022 / 11 / 18 / 1063487 / meta - large - language - model - ai - only - survived-three-days-gpt-3-science/ . Helber, Patrick, Benjamin Bischke, Andreas Dengel, and Damian Borth (2019). “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.” In: IEEE J...

work page 2019

Show all 30 references

[18]

Analogy as the core of cognition

Basic books New York. — ( 2001). “Analogy as the core of cognition.” In: The analogical mind: Perspectives from cognitive science, pp. 499–538. — ( 2023). Gödel, Escher, Bach, and AI. Accessed: October 29,

2001
[19]

Meta-Learning in Neural Networks: A Survey

The Atlantic. url: https : / / www . theatlantic . com / ideas / archive / 2023/07/godel-escher-bach-geb-ai/674589/ . Hospedales, T. M., A. Antoniou, P . Micaelli, and A. J. Storkey ( 2020). “Meta-Learning in Neural Networks: A Survey.” In: IEEE Trans- actions on Pattern Analy...

2023
[23]

Deep learning

url: https://twitter.com/ylecun/status/1718263303485501784. LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton ( 2015). “Deep learning.” In: nature 521.7553, pp. 436–444. LeCun, Yann, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D...

1989
[24]

Learning to generalize: Meta-learning for domain generalization

url: https://twitter.com/binarybits/status/ 1691558467146776957. Li, Da, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales (2018). “Learning to generalize: Meta-learning for domain generalization.” In: Thirty-Second AAAI Conference on Artificial Intelligence. Li, Kenneth (20...

2018 arXiv
[26]

Making ai forget you: Data deletion in machine learn- ing

Galilei, Galileo ( 2016). Sidereus Nuncius, Or The Sidereal Messenger . University of Chicago Press. url: http : / / people . reed . edu / ~wieting/mathematics537/SideriusNuncius.pdf. Ginart, Antonio, Melody Guan, Gregory Valiant, and James Y Zou (2019). “Making ai forget you:...

2019
[27]

Shaping Visual Representations with Language for Few-shot Classification

Mu, Jesse, Percy Liang, and Noah Goodman ( 2020). “Shaping Visual Representations with Language for Few-shot Classification.” In: ACL. Nadaraya, Elizbar A (1964). “On estimating regression.” In: Theory of Probability & Its Applications 9.1, pp. 141–142. Neumann, John von ( 192...

1964 arXiv
[29]

Why philosophers should care about compu- tational complexity

Table 12 with re- sults on the parameters fine-tuned on the two datasets reveals marginal improvements over the standard choice of k= 800 and p=8. This sug- gests that the ASIF model is relatively insensitive to the choice of these hyperparameters. Tuned on Parameters p,k CIFA...

2013
[30]

Mastering the game of Go without human knowledge

Silver, David et al. (2017). “Mastering the game of Go without human knowledge.” In: Nature 550.7676, pp. 354–359. Silver, David et al. ( 2018). “A general reinforcement learning algo- rithm that masters chess, shogi, and Go through self-play.” In: Science 362.6419, pp. 1140–1...

2017 arXiv
[31]

Discrete Key-Value Bottleneck

url: https://twitter.com/iamtrask/status/ 1695798588641538360. 104 bibliography Träuble, Frederik, Anirudh Goyal, Nasim Rahaman, Michael Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf (2022). “Discrete Key-Value Bottleneck.” In:arXiv preprint arXiv:2207.11240. V...

2022 arXiv
[32]

Multimodal Neurons in Artificial Neural Networks

Goh, Gabriel, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah ( 2021). “Multimodal Neurons in Artificial Neural Networks.” In: Distill. https://distill.pub/2021/multimodal-neurons. doi: 10.23915/distill. 00030. Golatkar,...

2020 arXiv
[47]

Generate & Rank: A Multi-task Framework for Math Word Problems

Shapiro, Ehud Y ( 1981). Inductive inference of theories from facts . Yale University, Department of Computer Science. Shen, Jianhao, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu ( 2021). “Generate & Rank: A Multi-task Framework for Math Word Problems.”...

1981 arXiv
[48]

Symbolic behaviour in artificial intelligence

ICML’16. New York, NY, USA: JMLR.org, 1842–1850. Santoro, Adam, Andrew Lampinen, Kory Mathewson, Timothy Lilli- crap, and David Raposo ( 2021). “Symbolic behaviour in artificial intelligence.” In: arXiv preprint arXiv:2102.03406. Schaeffer, Jonathan, Neil Burch, Yngvi Björnsso...

2007 arXiv
[88]

On the opportu- nities and risks of foundation models

Chicago, Illinois, USA: Association for Computing Machinery, 103–112. isbn: 0897912640. doi: 10.1145/62212.62222. url: https: //doi.org/10.1145/62212.62222. Bommasani, Rishi, Drew A Hudson, Ehsan Adeli, Russ Altman, Sim- ran Arora, Sydney von Arx, Michael S Bernstein, Jeannett...

2021
[162]

Language models are few-shot learners

Proceedings of Machine Learning Research. PMLR, pp. 2206–2240. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. ( 2020). “Language models are few-shot learners.” In...

2012 arXiv
[327]

LQF: Linear Quadratic Fine- Tuning

Achille, Alessandro, Aditya Golatkar, Avinash Ravichandran, Marzia Polito, and Stefano Soatto ( 2021). “LQF: Linear Quadratic Fine- Tuning.” In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15729–15739. Alayrac, Jean-Baptiste, J...

2023
[1087]

Ask & Ex- plore: Grounded Question Answering for Curiosity-Driven Ex- ploration

Kaur, Jivat Neet, Yiding Jiang, and Paul Pu Liang ( 2021). “Ask & Ex- plore: Grounded Question Answering for Curiosity-Driven Ex- ploration.” In: arXiv preprint arXiv: 2104.11902. Knuth, Donald E and Ronald W Moore ( 1975). “An analysis of alpha- beta pruning.” In: Artificial ...

2021 arXiv
[1704]

Errare humanum est? a pi- lot study to evaluate the human-likeness of a AI othello playing agent

04517 [cs.CL]. Lauletta, Enrico, Beatrice Biancardi, Antonio Norelli, Maurizio Mancini, and Alessandro Panconesi ( 2022). “Errare humanum est? a pi- lot study to evaluate the human-likeness of a AI othello playing agent.” In: Proceedings of the 22nd ACM International Conferenc...

2022
[1967]

Improving Language Models by Retrieving from Trillions of Tokens

Rochelle Park, N.J.: Hayden Book Co., Spartan Books. Borgeaud, Sebastian et al. ( 2022). “Improving Language Models by Retrieving from Trillions of Tokens.” In:International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Vol

2022
[2005]

World Models

seems inaccessible for computational systems that have never had any direct experience with the real world outside of text, as noted by Douglas Hofstadter (2023). However, it is important to acknowledge evidence pointing towards the capability of neural network-based AI models...

2023
[2021]

The DEIT models were pre-trained on a smaller dataset, the standard Imagenet 1k training set (Deng et al

( 86M, 768). The DEIT models were pre-trained on a smaller dataset, the standard Imagenet 1k training set (Deng et al. 2009), while VITb 16 was pretrained on Imagenet 21k (Ridnik et al. 2021). As text encoders, we used smaller versions of SentenceT (Reimers and Gurevych Nov. 2...

2009
[2023]

I do not know

showcases 64 are large language models sparks of artificial scientists ? the presence of an Othello world model in LLMs fine-tuned on Oth- ello moves. In this study, a variant of the GPT model was applied to predict legal moves in the board game Othello. The research uncov- er...

2022
[3137]

Chess, a Drosophila of reasoning

Kasparov, Garry ( 2018). “Chess, a Drosophila of reasoning.” In: Sci- ence (New York, N.Y.) 362.6419, p

2018

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [2]

Bootstrapping Parallel Anchors for Relative Representations

Citeseer. Cannistraci, Irene, Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, and Emanuele Rodolà ( 2023). “Bootstrapping Parallel Anchors for Relative Representations.” In: Tiny paper at ICLR

work page 2023

[2] [5]

a satellite image of [CLASS NAME]

achieved a zero-shot classification score of 29.4%. While significantly better than random chance, this modest performance is not surpris- B.2 additional details on the eurosat experiment . 89 Figure 28: ASIF performance does not saturate earlier with smaller encoders. Classification accuracy keeps growing without satu- rating but is lower for smaller mod...

work page 2021

[3] [6]

Beyond the Imitation Game: Quantify- ing and extrapolating the capabilities of language models

5.2.1 Big-Bench Big Bench serves as a comprehensive benchmark to evaluate and un- derstand the capabilities and limitations of contemporary language models (Srivastava et al. 2023). Developed collaboratively by 450 au- thors from 132 institutions, including myself, the benchmark consists of 204 diverse tasks covering areas such as linguistics, math, commo...

work page 1972

[4] [8]

DeepCoder: Learning to write programs

url: https://www.anthropic.com/index/introducing-claude. Balog, M, AL Gaunt, M Brockschmidt, S Nowozin, and D Tarlow (2017). “DeepCoder: Learning to write programs.” In: 5th Interna- tional Conference on Learning Representations, ICLR 2017-Conference Track Proceedings. Basu, Samyadeep, Phil Pope, and Soheil Feizi (2021). “Influence Func- tions in Deep Lea...

2017

[5] [9]

Relative rep- resentations enable zero-shot latent space communication

McGraw-hill New York. isbn: 0070428077. Mitchell, Tom M ( 1980). The need for biases in learning generalizations . Department of Computer Science, Laboratory for Computer Sci- ence Research. Moschella, Luca, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà ( 2022). “Relative rep- resentations enable zero-shot late...

work page 1980

[6] [13]

Performance analysis of google colaboratory as a tool for accelerating deep learning ap- plications

Carneiro, Tiago, Raul Victor Medeiros Da Nóbrega, Thiago Nepomu- ceno, Gui-Bin Bian, Victor Hugo C De Albuquerque, Pedro Pe- drosa Reboucas Filho, and X X ( 2018). “Performance analysis of google colaboratory as a tool for accelerating deep learning ap- plications.” In: IEEE Access 6, pp. 61677–61685. bibliography 95 Caron, Mathilde, Hugo Touvron, Ishan M...

arXiv 2018

[7] [14]

Dreamcoder: Growing gener- alizable, interpretable knowledge with wake-sleep bayesian pro- gram learning

Eco, Umberto (2000). Kant and the platypus: Essays on language and cog- nition. HMH. Ellis, Kevin, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum ( 2020). “Dreamcoder: Growing gener- alizable, interpretable knowledge with wake-sleep bayesian pro- gram learning.” In: arXi...

arXiv 2000

[8] [17]

Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

MIT Technology Review. url: https://www.technologyreview.com/ 2022 / 11 / 18 / 1063487 / meta - large - language - model - ai - only - survived-three-days-gpt-3-science/ . Helber, Patrick, Benjamin Bischke, Andreas Dengel, and Damian Borth (2019). “Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.” In: IEEE J...

work page 2019